Image processing apparatus and method and image pickup apparatus

ABSTRACT

The present invention allows an image to be processed corresponding to the mixture of a background image and an image of a moving object. A region specifying unit specifies a mixed region wherein foreground object components making up foreground objects and background object components making up background objects are mixed, and a non-mixed region configured of one of foreground object components making up foreground objects or background object components making up background objects, and outputs region information corresponding to the results of specifying. A region processing unit  5001  processes input images for each of the regions specified by the region information. The present invention can be applied to image processing devices for processing images.

TECHNICAL FIELD

[0001] The present invention relates to an image processing device andmethod, and an image-taking device, and particularly relates to an imageprocessing device and method, and an image-taking device, which takeinto consideration difference between signals detected by sensors andthe real world.

BACKGROUND ART

[0002] One type of processing for generating images with higherresolution based on input images, is class classification adaptationprocessing. An example of class classification adaptation processing isprocessing wherein coefficients used in processing for generating imageswith higher resolution are generated beforehand, in the spatialdirection, and images are generated with higher resolution in thespatial direction based on the generated coefficients.

[0003]FIG. 1 is a block diagram illustrating the configuration of aconventional image processing device for generating coefficients used inclass classification adaptation processing for generating HD (HighDefinition) images from SD (Standard Definition) images.

[0004] Frame memory 11 stores input images, which are HD images, inincrements of frames. The frame memory 11 supplies the stored HD imagesto a weighted averaging unit 12 and a corresponding pixel obtaining unit16.

[0005] The weighted averaging unit 12 performs one-quarter weightedaveraging on the HD images stored in the frame memory 11, generates SDimages, and supplies the generated SD images to the frame memory 13.

[0006] The frame memory 13 stores the SD images supplied from theweighted averaging unit 12 in increments of frames, and supplies thestored SD images to a class classification unit 14 and prediction tapobtaining unit 15.

[0007] The class classification unit 14 is configured of a class tapobtaining unit 21 and a waveform classification unit 22, and performsclass classification of pixels of interest which are the pixel ofinterest in the SD images stored in the frame memory 13. The class tapobtaining unit 21 obtains a predetermined number of class taps which arepixels of the SD image corresponding to the pixel of interest from theframe memory 13, and supplies the obtained class taps to the waveformclassification unit 22.

[0008]FIG. 2 is a diagram explaining the class taps obtained by theclass tap obtaining unit 21. As shown in FIG. 2, the class tap obtainingunit 21 obtains eleven class taps at predetermined positions.

[0009] The waveform classification unit 22 classifies the pixel ofinterest into one class out of multiple classes, based on the classtaps, and supplies a class No. corresponding to the classified class, tothe prediction tap obtaining unit 15. The waveform classification unit22 classifies the pixel of interest into one class out of 2048 classes,based on the eleven class taps.

[0010] The prediction tap obtaining unit 15 obtains a predeterminednumber of prediction taps which are pixels of the SD image,corresponding to the classified class from the frame memory 13, based onthe class No., and supplies the obtained prediction taps and class Nos.to a corresponding pixel obtaining unit 16.

[0011]FIG. 3 is a diagram explaining prediction taps which theprediction tap obtaining unit 15 obtains. As shown in FIG. 3, theprediction tap obtaining unit 15 obtains nine prediction taps atpredetermined locations.

[0012] The corresponding pixel obtaining unit 16 obtains, from the framememory 11, pixels of the HD image corresponding to the pixel values tobe predicted, based on the prediction taps and the class Nos., andsupplies the prediction taps, class Nos., and the pixels of the HD imagecorresponding to the obtained pixel values to be predicted, to a normalequation generating unit 17.

[0013] The normal equation generating unit 17 generates normal equationscorresponding to relationships between prediction taps and pixel valuesto be predicted, corresponding to the classes, based on the predictiontaps, class Nos., and the obtained pixel values to be predicted, andsupplies the generated normal equations corresponding to the classes, toa coefficient calculation unit 18.

[0014] The coefficient calculation unit 18 solves the normal equationsupplied from the normal equation generating unit 17, calculatescoefficient sets corresponding to each class, and supplies thecalculated coefficient sets to coefficient set memory 19, along with theclass Nos.

[0015] The coefficient set memory 19 stores the calculated coefficientsets corresponding to the classes, based on the class Nos.

[0016]FIG. 4 is a diagram explaining an overview of class classificationadaptation processing. In class classification adaptation processing, atutor image which is an HD image is used to generate a corresponding SDimage, by one-quarter weighted average processing. The generated SDimage is called a student image.

[0017] Next, a coefficient set for generating an HD image from the SDimage is generated, based on the tutor image which is the HD image andthe student image which is the corresponding SD image. The coefficientset is configured of coefficients for generating an HD image from an SDimage, by linear prediction and the like.

[0018] A quadruple-density image is generated from the coefficients setthus generated and the SD image, by linear prediction and the like. Theprocessing for generating an image or the like with higher density, froma coefficient set and an input image, is also called mapping.

[0019] SNR comparison, or visual qualitative evaluation is performed,based on the generated quadruple-density image and a corresponding HDimage.

[0020] A coefficient set generated from a particular tutor image andcorresponding student image is called a self coefficient set of theparticular tutor image and corresponding student image. Mapping usingthe self coefficient set is called self mapping. A coefficient setgenerated from multiple other tutor images and corresponding studentimages is called a cross coefficient set.

[0021] On the other hand, with images obtained by a video camera takinga foreground subject which moves across a predetermined stationarybackground, movement blurring occurs in the event that the speed ofmovement of the object is relatively fast, and mixing of the foregroundand background occurs.

[0022] With conventional class classification adaptation processing, oneset of coefficients is generated for all of the foreground, background,and portions where mixing between the foreground and background occurs,by learning processing such as described above, and mapping processingis executed based on the coefficient set.

[0023] The conventional learning processing for generating coefficientsused in the processing for generating HD images from SD images will bedescribed, with reference to the flowchart shown in FIG. 6. In Step S11,an image processing device judges whether or not there are anyunprocessed pixels in the student image, and in the event that judgmentis made that there are unprocessed pixels in the student image, the flowproceeds to Step S12, and pixels of interest are obtained from thestudent image, in order of raster scan.

[0024] In Step S13, the class tap obtaining unit 21 of the classclassification unit 14 obtains a class tap corresponding to the pixel ofinterest, from the student image stored in the frame memory 13. In StepS14, the waveform classification unit 22 of the class classificationunit 14 performs class classification of the pixel of interest, based onthe class tap. In Step S15, the prediction tap obtaining unit 15 obtainsa prediction tap corresponding to the pixel of interest from the studentimage stored in the frame memory 13, based on the class into whichclassification has been made.

[0025] In Step S16, the corresponding pixel obtaining unit 16 obtains apixel corresponding to a pixel value to be predicted, from tutor datastored in the frame memory 11, based on the class into whichclassification has been made.

[0026] In Step S17, the normal equation generating unit 17 adds a pixelvalue of a pixel corresponding to the prediction tap and pixel value tobe predicted to the matrix for each class, based on the class into whichclassification has been made, the flow returns to Step S11, and theimage processing device repeats judgment regarding whether or not thereare any unprocessed pixels. The matrixes for each class to which thepixel value of a pixel corresponding to the prediction tap and pixelvalue to be predicted are added, correspond to the normal equation forcalculating coefficients for each class.

[0027] In the event that judgment is made in Step S11 that there are nounprocessed pixels in the student image, the flow proceeds to Step S18,wherein the normal equation generating unit 17 supplies the matrix foreach class wherein a pixel value of a pixel corresponding to theprediction tap and pixel value to be predicted has been set, to thecoefficient calculation unit 18. The coefficient calculation unit 18solves the matrix for each class wherein a pixel value of a pixelcorresponding to the prediction tap and pixel value to be predicted hasbeen set, and calculates a coefficient set for each class.

[0028] In Step S19, the coefficient calculation unit 18 outputs thecoefficient for each class that has been calculated, to the coefficientset memory 19. The coefficient set memory 19 stores a coefficient setfor each class, and the processing ends.

[0029]FIG. 7 is a block diagram illustrating the configuration of aconventional image processing device for generating HD images from SDimages, by class classification adaptation processing.

[0030] Frame memory 31 stores input images which are SD images, inincrements of frames. The frame memory 31 supplies the stored SD imagesto a mapping unit 32.

[0031] The SD images input to the mapping unit 32 are supplied to aclass classification unit 41 and a prediction tap obtaining unit 42.

[0032] The class classification unit 41 is configured of a class tapobtaining unit 51 and a waveform classification unit 52, and performsclass classification of pixels of interest which are the pixel ofinterest in the SD images stored in the frame memory 31. The class tapobtaining unit 51 obtains from the frame memory 31 a predeterminednumber of class taps corresponding to the pixel of interest, andsupplies the obtained class taps to the waveform classification unit 52.

[0033] The waveform classification unit 52 classifies the pixel ofinterest into one class out of multiple classes, based on the classtaps, and supplies a class No. corresponding to the classified class, tothe prediction tap obtaining unit 42.

[0034] The prediction tap obtaining unit 42 obtains from the input imagestored in the frame memory 31 a predetermined number of prediction tapscorresponding to the classified class, based on the class No., andsupplies the obtained prediction taps and class Nos. to a predictioncomputation unit 43.

[0035] The prediction computation unit 43 obtains coefficient setscorresponding to classes from the coefficient sets stored in coefficientset memory 33, based on the class No. The prediction computation unit 43predicts pixel values of predicted images by linear prediction, based oncoefficient sets corresponding to classes, and prediction taps. Theprediction computation unit 43 supplies the predicted pixel values toframe memory 34.

[0036] The frame memory 34 stores predicted pixel values supplied fromthe prediction computation unit 43, and outputs an HD image wherein thepredicted pixel values have been set.

[0037]FIG. 8 is a diagram illustrating the pixel values of the inputimage, and the pixel values of the output image generated by classclassification adaptation processing. In FIG. 8, the white squaresindicate input signals, and the solid circles indicate output signals.As shown in FIG. 8, the image generated by the class classificationadaptation processing contains waveforms lost in the bandwidthrestriction of the SD image. In this sense, it can be said thatprocessing for generating an image with higher resolution by the classclassification adaptation processing creates resolution.

[0038] The conventional processing for creating images, for generatingHD images from SD image with an image processing device which executesclass classification adaptation processing, will be described withreference to the flowchart in FIG. 9.

[0039] In Step S31, the image processing device judges whether or notthere are any unprocessed pixels in the input image, and in the eventthat judgment is made that there are unprocessed pixels in the inputimage, the flow proceeds to Step S32, where the mapping unit 32 obtainsa coefficient set stored in the coefficient set memory 33. In Step S33,the image processing device obtains pixels of interest from the inputimage in raster scan order.

[0040] In Step S34, the class tap obtaining unit 51 of the classclassification unit 41 obtains a class tap corresponding to the pixel ofinterest, from the input image stored in the frame memory 31. In StepS35, the waveform classification unit 52 of the class classificationunit 41 performs class classification of the pixel of interest into oneclass, based on the class tap.

[0041] In Step S36, the prediction tap obtaining unit 42 obtains aprediction tap corresponding to the pixel of interest from the inputimage stored in the frame memory 31, based on the class into whichclassification has been made.

[0042] In Step S37, the prediction computation unit 43 obtains a pixelvalue of a predicted image by linear prediction, based on thecoefficient set corresponding to the class into which classification hasbeen made, and the prediction tap.

[0043] In Step S38, the prediction computation unit 43 outputs thepredicted pixel value to the frame memory 34. The frame memory 34 storesthe pixel value supplied from the prediction computation unit 43. Theprocedures return to Step S31, and repeats judgement regarding whetheror not there are any unprocessed pixels.

[0044] In the event that judgment is made in Step S31 that there are nounprocessed pixels in an input image, the flow proceeds to Step S39,where the frame memory 34 outputs the stored predicted image whereinpredicted values are set, and the processing ends.

[0045] Also, processing for edge enhancing of images is widely used asprocessing for raising the sense of resolution of the image.

[0046] However, in the event that objects move in front of stillbackgrounds, movement blurring occurs not only due to mixture of themoving object images itself, but also due to mixture of the movingobject images and the background images. Conventionally, processingimages corresponding to the mixing of the background image and the imageof the moving object had not been given thought.

[0047] Also, applying edge enhancing processing to image containingmovement blurring has resulted in unnatural images at times. Setting thedegree of edge enhancing lower so that such unnatural images do notoccur has resulted in insufficient improvement in sense of resolution ofthe image.

DISCLOSURE OF INVENTION

[0048] The present invention has been made in light of the above, and itis an object thereof to enable processing of images corresponding to themixing of background images and images of the moving objects.

[0049] Also, it is another object thereof to sufficiently raise sense ofresolution without making images containing movement blurring intounnatural images.

[0050] A first image processing device according to the presentinvention comprises: region specifying means for specifying, based onthe input image data, a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and processing means for processing the input image data foreach region specified by the region specifying information.

[0051] The processing means may decide a class corresponding to eachpiece of pixel data of the input image data, corresponding to the regionspecifying information.

[0052] The processing means may enhance the edges of the input imagedata, corresponding to the region specifying information.

[0053] The processing means may process the pixel data of at least oneregion of the mixed region and the non-mixed region.

[0054] The region specifying means may further specify a coveredbackground region and an uncovered background region, and output regionspecifying information corresponding to the results of specifying, withthe processing means further processing the input image data for each ofcovered background region and uncovered background region.

[0055] The processing means may generate coefficients used in classclassification adaptation processing, for each region specified by theregion specifying information.

[0056] The processing means may generate output image data by classclassification adaptation processing, for each region specified by theregion specifying information.

[0057] The processing means may enhance the edges of the input imagedata, for each region specified by the region specifying information.

[0058] A first image processing method according to the presentinvention comprises: a region specifying step for specifying, based onthe input image data, a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and a processing step for processing the input image datafor each region specified by the region specifying information.

[0059] In the processing step, a class corresponding to each piece ofpixel data of the input image data may be decided, corresponding to theregion specifying information.

[0060] In the processing step, the edges of the input image data may beenhanced, corresponding to the region specifying information.

[0061] In the processing step, the pixel data of at least one region ofthe mixed region and the non-mixed region may be processed.

[0062] In the region specifying step, a covered background region and anuncovered background region may be further specified, with regionspecifying information being output corresponding to the results ofspecifying; and in the processing step, the input image data for each ofcovered background region and uncovered background region may be furtherprocessed.

[0063] In the processing step, coefficients used in class classificationadaptation processing may be generated for each region specified by theregion specifying information.

[0064] In the processing step, output image data may be generated byclass classification adaptation processing for each region specified bythe region specifying information.

[0065] In the processing step, the edges of the input image data may beenhanced for each region specified by the region specifying information.

[0066] A program in a first recording medium according to the presentinvention comprises: a region specifying step for specifying, based onthe input image data, a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and a processing step for processing the input image datafor each region specified by the region specifying information.

[0067] In the processing step, a class corresponding to each piece ofpixel data of the input image data may be decided, corresponding to theregion specifying information.

[0068] In the processing step, the edges of the input image data may beenhanced, corresponding to the region specifying information.

[0069] In the processing step, the pixel data of at least one region ofthe mixed region and the non-mixed region may be processed.

[0070] In the region specifying step, a covered background region and anuncovered background region may be further specified, with regionspecifying information being output corresponding to the results ofspecifying; and in the processing step, the input image data for each ofcovered background region and uncovered background region may be furtherprocessed.

[0071] In the processing step, coefficients used in class classificationadaptation processing may be generated for each region specified by theregion specifying information.

[0072] In the processing step, output image data may be generated byclass classification adaptation processing for each region specified bythe region specifying information.

[0073] In the processing step, the edges of the input image data may beenhanced for each region specified by the region specifying information.

[0074] A first program according to the present invention causes acomputer to execute: a region specifying step for specifying, based onthe input image data, a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and a processing step for processing the input image datafor each region specified by the region specifying information.

[0075] In the processing step, a class corresponding to each piece ofpixel data of the input image data may be decided, corresponding to theregion specifying information.

[0076] In the processing step, the edges of the input image data may beenhanced, corresponding to the region specifying information.

[0077] In the processing step, the pixel data of at least one region ofthe mixed region and the non-mixed region may be processed.

[0078] In the region specifying step, a covered background region and anuncovered background region may be further specified, with regionspecifying information being output corresponding to the results ofspecifying; and in the processing step, the input image data for each ofcovered background region and uncovered background region may be furtherprocessed.

[0079] In the processing step, coefficients used in class classificationadaptation processing may be generated for each region specified by theregion specifying information.

[0080] In the processing step, output image data may be generated byclass classification adaptation processing for each region specified bythe region specifying information.

[0081] In the processing step, the edges of the input image data may beenhanced for each region specified by the region specifying information.

[0082] A first image-taking device according to the present inventioncomprises: image-taking means for outputting a subject image taken by animage-taking device having a predetermined number of pixels havingtime-integration effects as image data made up of a predetermined numberof pieces of pixel data; region specifying means for specifying, basedon the image data, a mixed region made up of a mixture of a foregroundobject component configuring foreground objects and a background objectcomponent configuring background objects, and a non-mixed region made upof one of a foreground region made up of the foreground object componentand a background region made up of a background object componentconfiguring the background objects, and outputting region specifyinginformation corresponding to the results of specifying; and processingmeans for processing the image data for each region specified by theregion specifying information.

[0083] The processing means may decide a class corresponding to eachpiece of pixel data of the input image data, corresponding to the regionspecifying information.

[0084] The processing means may enhance the edges of the input imagedata, corresponding to the region specifying information.

[0085] The processing means may process the pixel data of at least oneregion of the mixed region and the non-mixed region.

[0086] The region specifying means may further specify a coveredbackground region and an uncovered background region, and output regionspecifying information corresponding to the results of specifying; andthe processing means may further process the input image data for eachof covered background region and uncovered background region.

[0087] The processing means may generate coefficients used in classclassification adaptation processing, for each region specified by theregion specifying information.

[0088] The processing means may generate output image data by classclassification adaptation processing, for each region specified by theregion specifying information.

[0089] The processing means may enhance the edges of the input imagedata, for each region specified by the region specifying information.

[0090] A second image processing device according to the presentinvention comprises: region specifying means for specifying, based onthe input image data, a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and class deciding means for determining classescorresponding to each set of pixel data of the input image data,corresponding to the region specifying information.

[0091] The class deciding means may decide a class corresponding to thepixel data of only regions which are a portion of the mixed region, theforeground region, and the background region.

[0092] The image processing device may further comprise generating meansfor processing the pixel data of the input image data corresponding tothe classes that have been decided, and generating coefficients used inclass classification adaptation processing.

[0093] The image processing device may further comprise converting meansfor processing the pixel data of the input image data based on acoefficient for each of the classes, corresponding to the classes thathave been decided, and converting the input image data into output imagedata.

[0094] The region specifying means may further specify a coveredbackground region and an uncovered background region, and output theregion specifying information corresponding to the results ofspecifying; and the class deciding means may further decide the classescorresponding to the pixel data of the input image data, correspondingto the covered background region or the uncovered background region thathave been specified.

[0095] A second image processing method according to the presentinvention comprises: a region specifying step for specifying, based onthe input image data, a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and a class deciding step for determining classescorresponding to each set of pixel data of the input image data,corresponding to the region specifying information.

[0096] In the class deciding step, a class corresponding to the pixeldata of only regions which are a portion of the mixed region, theforeground region, and the background region, may be decided.

[0097] The image processing method may further comprise a generatingstep for processing the pixel data of the input image data correspondingto the classes that have been decided, and generating coefficients usedin class classification adaptation processing.

[0098] The image processing method may further comprise a convertingstep for processing the pixel data of the input image data based on acoefficient for each of the classes, corresponding to the classes thathave been decided, and converting the input image data into output imagedata.

[0099] In the region specifying step, a covered background region and anuncovered background region may be further specified, with the regionspecifying information being output corresponding to the results ofspecifying; and in the class deciding step, the classes corresponding tothe pixel data of the input image data may be decided corresponding tothe covered background region or the uncovered background region thathave been specified.

[0100] A program in a second recording medium according to the presentinvention comprises: a region specifying step for specifying, based onthe input image data, a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and a class deciding step for determining classescorresponding to each set of pixel data of the input image data,corresponding to the region specifying information.

[0101] In the class deciding step, a class corresponding to the pixeldata of only regions which are a portion of the mixed region, theforeground region, and the background region, may be decided.

[0102] The program in the recording medium may further comprise agenerating step for processing the pixel data of the input image datacorresponding to the classes that have been decided, and generatingcoefficients used in class classification adaptation processing.

[0103] The program in the recording medium may further comprise aconverting step for processing the pixel data of the input image databased on a coefficient for each of the classes, corresponding to theclasses that have been decided, and converting the input image data intooutput image data.

[0104] In the region specifying step, a covered background region and anuncovered background region may be further specified, with the regionspecifying information being output corresponding to the results ofspecifying; and in the class deciding step, the classes corresponding tothe pixel data of the input image data may be decided corresponding tothe covered background region or the uncovered background region thathave been specified.

[0105] A second program according to the present invention causes acomputer to execute: a region specifying step for specifying, based onthe input image data, a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and a class deciding step for determining classescorresponding to each set of pixel data of the input image data,corresponding to the region specifying information.

[0106] In the class deciding step, a class corresponding to the pixeldata of only regions which are a portion of the mixed region, theforeground region, and the background region, may be decided.

[0107] The program may further comprise a generating step for processingthe pixel data of the input image data corresponding to the classes thathave been decided, and generating coefficients used in classclassification adaptation processing.

[0108] The program may further comprise a converting step for processingthe pixel data of the input image data based on a coefficient for eachof the classes, corresponding to the classes that have been decided, andconverting the input image data into output image data.

[0109] In the region specifying step, a covered background region and anuncovered background region may be further specified, with the regionspecifying information being output corresponding to the results ofspecifying; and in the class deciding step, the classes corresponding tothe pixel data of the input image data may be decided corresponding tothe covered background region or the uncovered background region thathave been specified.

[0110] A second image-taking device according to the present inventioncomprises: image-taking means for outputting a subject image taken by animage-taking device having a predetermined number of pixels havingtime-integration effects as taken image data made up of a predeterminednumber of pieces of pixel data; region specifying means for specifying,based on the taken image data, a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and class deciding means for determining classescorresponding to each set of pixel data of the taken image data,corresponding to the region specifying information.

[0111] The class deciding means may decide a class corresponding to thepixel data of only regions which are a portion of the mixed region, theforeground region, and the background region.

[0112] The image-taking device may further comprise generating means forprocessing the pixel data of the input image data corresponding to theclasses that have been decided, and generating coefficients used inclass classification adaptation processing.

[0113] The image-taking device may further comprise converting means forprocessing the pixel data of the input image data based on a coefficientfor each of the classes, corresponding to the classes that have beendecided, and converting the input image data into output image data.

[0114] The region specifying means may further specify a coveredbackground region and an uncovered background region, and output theregion specifying information corresponding to the results ofspecifying; and the class deciding means may decide the classescorresponding to the pixel data of the input image data, correspondingto the covered background region or the uncovered background region thathave been specified.

[0115] A third image processing device according to the presentinvention comprises: region specifying means for specifying, based onthe input image data, a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and edge enhancing means for enhancing the edges of theinput image data, corresponding to the region specifying information.

[0116] The region specifying means may further specify a coveredbackground region and an uncovered background region, and output theregion specifying information corresponding to the results ofspecifying; and the edge enhancing means may enhance the edges of theinput image data, corresponding to the covered background region or theuncovered background region that have been specified.

[0117] A third image processing method according to the presentinvention comprises: a region specifying step for specifying, based onthe input image data, a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and an edge enhancing step for enhancing the edges of theinput image data, corresponding to the region specifying information.

[0118] In the region specifying step, a covered background region and anuncovered background region may be further specified, with the regionspecifying information being output corresponding to the results ofspecifying; and in the edge enhancing step, the edges of the input imagedata may be enhanced corresponding to the covered background region orthe uncovered background region that have been specified.

[0119] A program in a third recording medium comprises: a regionspecifying step for specifying, based on the input image data, a mixedregion made up of a mixture of a foreground object component configuringforeground objects and a background object component configuringbackground objects, and a non-mixed region made up of one of aforeground region made up of the foreground object component and abackground region made up of a background object component configuringthe background objects, and outputting region specifying informationcorresponding to the results of specifying; and an edge enhancing stepfor enhancing the edges of the input image data, corresponding to theregion specifying information.

[0120] In the region specifying step, a covered background region and anuncovered background region may be further specified, with the regionspecifying information being output corresponding to the results ofspecifying; and in the edge enhancing step, the edges of the input imagedata may be enhanced corresponding to the covered background region orthe uncovered background region that have been specified.

[0121] A third program according to the present invention causes acomputer to execute: a region specifying step for specifying, based onthe input image data, a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and an edge enhancing step for enhancing the edges of theinput image data, corresponding to the region specifying information.

[0122] In the region specifying step, a covered background region and anuncovered background region may be further specified, with the regionspecifying information being output corresponding to the results ofspecifying; and in the edge enhancing step, the edges of the input imagedata may be enhanced corresponding to the covered background region orthe uncovered background region that have been specified.

[0123] A third image-taking device according to the present inventioncomprises: image-taking means for outputting a subject image taken by animage-taking device having a predetermined number of pixels havingtime-integration effects as image data made up of a predetermined numberof pieces of pixel data; region specifying means for specifying, basedon the image data, a mixed region made up of a mixture of a foregroundobject component configuring foreground objects and a background objectcomponent configuring background objects, and a non-mixed region made upof one of a foreground region made up of the foreground object componentand a background region made up of a background object componentconfiguring the background objects, and outputting region specifyinginformation corresponding to the results of specifying; and edgeenhancing means for enhancing the edges of the image data, correspondingto the region specifying information.

[0124] The region specifying means may further specify a coveredbackground region and an uncovered background region, and output theregion specifying information corresponding to the results ofspecifying; and the edge enhancing means may enhance the edges of theimage data, corresponding to the covered background region or theuncovered background region that have been specified.

[0125] A fourth image processing device according to the presentinvention comprises: region specifying means for specifying, based onthe input image data, at least one of a mixed region made up of amixture of a foreground object component configuring foreground objectsand a background object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and processing means for processing the pixel data for atleast one region of the mixed region and the non-mixed region.

[0126] The processing means may process one region specified by theregion specifying means with a method different from a method forprocessing the other region.

[0127] The region specifying means may further specify the foregroundregion and the background region, and output the region specifyinginformation corresponding to the results of specifying.

[0128] The region specifying means may further specify a coveredbackground region and an uncovered background region, and output regionspecifying information corresponding to the results of specifying.

[0129] The image processing device may further comprise separating meansfor separating the pixel data of the mixed region into the foregroundobject component and the background object component, based on theregion specifying information, with the processing means processing atleast one of the foreground object component and the background objectcomponent.

[0130] A fourth image processing method according to the presentinvention comprises: a region specifying step for specifying, based onthe input image data, at least one of a mixed region made up of amixture of a foreground object component configuring foreground objectsand a background object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and a processing step for processing the pixel data for atleast one region of the mixed region and the non-mixed region.

[0131] In the processing step, one region specified by the processing inthe region specifying step may be processed with a method different froma method for processing the other region.

[0132] In the region specifying step, the foreground region and thebackground region may be further specified, with the region specifyinginformation being output corresponding to the results of specifying.

[0133] In the region specifying step, a covered background region and anuncovered background region may be further specified, with the regionspecifying information being output corresponding to the results ofspecifying.

[0134] The image processing method may further comprise a separatingstep for separating the pixel data of the mixed region into theforeground object component and the background object component, basedon the region specifying information; and in the processing step, atleast one of the foreground object component and the background objectcomponent may be processed.

[0135] A program in a fourth recording medium according to the presentinvention comprises: a region specifying step for specifying, based onthe input image data, at least one of a mixed region made up of amixture of a foreground object component configuring foreground objectsand a background object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and a processing step for processing the pixel data for atleast one region of the mixed region and the non-mixed region.

[0136] In the processing step, one region specified by the processing inthe region specifying step may be processed with a method different froma method for processing the other region.

[0137] In the region specifying step, the foreground region and thebackground region may be further specified, with the region specifyinginformation being output corresponding to the results of specifying.

[0138] In the region specifying step, a covered background region and anuncovered background region may be further specified, with the regionspecifying information being output corresponding to the results ofspecifying.

[0139] The program in the recording medium may further comprise aseparating step for separating the pixel data of the mixed region intothe foreground object component and the background object component,based on the region specifying information; and in the processing step,at least one of the foreground object component and the backgroundobject component may be processed.

[0140] A fourth program according to the present invention causes acomputer to execute: a region specifying step for specifying, based onthe input image data, at least one of a mixed region made up of amixture of a foreground object component configuring foreground objectsand a background object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and a processing step for processing the pixel data for atleast one region of the mixed region and the non-mixed region.

[0141] In the processing step, one region specified by the processing inthe region specifying step may be processed with a method different froma method for processing the other region.

[0142] In the region specifying step, the foreground region and thebackground region may be further specified, with the region specifyinginformation being output corresponding to the results of specifying.

[0143] In the region specifying step, a covered background region and anuncovered background region may be further specified, with the regionspecifying information being output corresponding to the results ofspecifying.

[0144] The program may further comprise a separating step for separatingthe pixel data of the mixed region into the foreground object componentand the background object component, based on the region specifyinginformation; and in the processing step, at least one of the foregroundobject component and the background object component may be processed.

[0145] A fourth image-taking device according to the present inventioncomprises: image-taking means for outputting a subject image taken by animage-taking device having a predetermined number of pixels havingtime-integration effects as image data made up of a predetermined numberof pieces of pixel data; region specifying means for specifying, basedon the image data, at least one of a mixed region made up of a mixtureof a foreground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, andoutputting region specifying information corresponding to the results ofspecifying; and processing means for processing the pixel data for atleast one region of the mixed region and the non-mixed region.

[0146] The processing means may process one region specified by theregion specifying means with a method different from a method forprocessing the other region.

[0147] The region specifying means may further specify the foregroundregion and the background region, and output the region specifyinginformation corresponding to the results of specifying.

[0148] The region specifying means may further specify a coveredbackground region and an uncovered background region, and output theregion specifying information corresponding to the results ofspecifying.

[0149] The image-taking device may further comprise separating means forseparating the pixel data of the mixed region into the foreground objectcomponent and the background object component, based on the regionspecifying information; with the processing means processing at leastone of the foreground object component and the background objectcomponent.

[0150] A mixed region made up of a mixture of a foreground objectcomponent configuring foreground objects and a background objectcomponent configuring background objects, and a non-mixed region made upof one of a foreground region made up of the foreground object componentand a background region made up of a background object componentconfiguring the background objects, are specified based on input imagedata, region specifying information corresponding to the results ofspecifying is output, and the input image data for each region specifiedby the region specifying information is processed.

[0151] A mixed region made up of a mixture of a foreground objectcomponent configuring foreground objects and a background objectcomponent configuring background objects, and a non-mixed region made upof one of a foreground region made up of the foreground object componentand a background region made up of a background object componentconfiguring the background objects, are specified based on input imagedata, region specifying information corresponding to the results ofspecifying is output, and classes corresponding to each set of pixeldata of the input image data are determined, corresponding to the regionspecifying information.

[0152] Thus, images can be processed corresponding to the mixing ofbackground images and moving objects.

[0153] A mixed region made up of a mixture of a foreground objectcomponent configuring foreground objects and a background objectcomponent configuring background objects, and a non-mixed region made upof one of a foreground region made up of the foreground object componentand a background region made up of a background object componentconfiguring the background objects, are specified based on input imagedata, outputting region specifying information corresponding to theresults of specifying is output, and the edges of the input image dataare enhanced, corresponding to the region specifying information.

[0154] Thus, the sense-of-resolution of images containing movementblurring can be sufficiently raised, without forming unnatural images.

[0155] At least one of a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of theforeground object component and a background region made up of abackground object component configuring the background objects, isspecified based on input image data, region specifying informationcorresponding to the results of specifying is output, and pixel data isprocessed for at least one region of the mixed region and the non-mixedregion.

[0156] Thus, images can be processed corresponding to the mixing ofbackground images and moving objects.

BRIEF DESCRIPTION OF THE DRAWINGS

[0157]FIG. 1 is a block diagram illustrating the configuration of aconventional image processing device.

[0158]FIG. 2 is a diagram explaining class taps.

[0159]FIG. 3 is a diagram explaining prediction taps.

[0160]FIG. 4 is a diagram describing the overview of classclassification adaptation processing.

[0161]FIG. 5 is a diagram explaining conventional coefficient sets.

[0162]FIG. 6 is a flowchart explaining conventional learning processing.

[0163]FIG. 7 is a block diagram illustrating the configuration of aconventional image processing device.

[0164]FIG. 8 is a diagram illustrating pixel values of an output imagegenerated by pixel values of an input image, and class classificationadaptation processing.

[0165]FIG. 9 is a flowchart explaining conventional processing forcreating images.

[0166]FIG. 10 is a block diagram illustrating the configuration of anembodiment of an image processing device according to the presentinvention.

[0167]FIG. 11 is a block diagram illustrating a configuration of animage processing device.

[0168]FIG. 12 is a diagram describing image-taking by a sensor.

[0169]FIG. 13 is a diagram describing an arrangement of pixels.

[0170]FIG. 14 is a diagram describing operation of a detecting device.

[0171]FIG. 15A is a diagram describing an image obtained by taking animage of an object corresponding to a moving foreground, and an objectcorresponding to a still background.

[0172]FIG. 15B is a diagram describing a model corresponding to an imageobtained by taking an image of an object corresponding to a movingforeground, and an object corresponding to a still background.

[0173]FIG. 16 is a diagram describing background region, foregroundregion, mixed region, covered background region, and uncoveredbackground region.

[0174]FIG. 17 is a model diagram which develops, over the timedirection, the pixel values of pixels adjacently arrayed in one row, inan image wherein an object corresponding to a still foreground and anobject corresponding to a still background are subjected toimage-taking.

[0175]FIG. 18 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0176]FIG. 19 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0177]FIG. 20 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0178]FIG. 21 is a diagram illustrating an example of extracting pixelsof the foreground region, background region, and mixed region.

[0179]FIG. 22 is a diagram illustrating how pixels correspond to a modelwherein pixel values are developed over the time direction.

[0180]FIG. 23 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0181]FIG. 24 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0182]FIG. 25 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0183]FIG. 26 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0184]FIG. 27 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0185]FIG. 28 is a diagram illustrating the correlation between adivided image, and a model diagram wherein the pixel values of pixelsare developed over the time direction.

[0186]FIG. 29 is a model diagram illustrating the correlation between adivided image, and a model diagram wherein the pixel values of pixelsare developed over the time direction.

[0187]FIG. 30 is a diagram illustrating an example of a divided image.

[0188]FIG. 31 is a diagram illustrating an example of a divided image.

[0189]FIG. 32 is a diagram illustrating the correlation between an imagewherein movement blurring has been removed, and a model diagram whereinthe pixel values of pixels are developed over the time direction.

[0190]FIG. 33 is a diagram describing processing of the image processingdevice according to the present invention.

[0191]FIG. 34 is a flowchart explaining image processing with the imageprocessing device according to the present invention.

[0192]FIG. 35 is a block diagram illustrating the configuration of theregion specifying unit 103.

[0193]FIG. 36 is a diagram describing an image wherein an objectcorresponding to the foreground is moving.

[0194]FIG. 37 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0195]FIG. 38 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0196]FIG. 39 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0197]FIG. 40 is a diagram describing conditions for region judgment.

[0198]FIG. 41A is a diagram illustrating an example of the results ofregion specification made by the region specifying unit 103.

[0199]FIG. 41B is a diagram illustrating an example of the results ofregion specification made by the region specifying unit 103.

[0200]FIG. 41C is a diagram illustrating an example of the results ofregion specification made by the region specifying unit 103.

[0201]FIG. 41D is a diagram illustrating an example of the results ofregion specification made by the region specifying unit 103.

[0202]FIG. 42 is a diagram illustrating an example of the results ofregion specification made by the region specifying unit 103.

[0203]FIG. 43 is a flowchart describing processing for regionspecifying.

[0204]FIG. 44 is a block diagram illustrating another configuration ofthe region specifying unit 103.

[0205]FIG. 45 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0206]FIG. 46 is a diagram illustrating an example of a backgroundimage.

[0207]FIG. 47 is a block diagram illustrating the configuration of abinary object image extracting unit 302.

[0208]FIG. 48A is a diagram describing calculating of correlationvalues.

[0209]FIG. 48B is a diagram describing calculating of correlationvalues.

[0210]FIG. 49A is a diagram describing calculating of correlationvalues.

[0211]FIG. 49B is a diagram describing calculating of correlationvalues.

[0212]FIG. 50 is a diagram illustrating an example of a binary objectimage.

[0213]FIG. 51 is a block diagram illustrating the configuration of atime change detecting unit 303.

[0214]FIG. 52 is a diagram describing judgment of a region judgment unit342.

[0215]FIG. 53 is a diagram illustrating an example of judgment made bythe time change detecting unit 303.

[0216]FIG. 54 is a flowchart describing processing for regionspecification by the region judgment unit 103.

[0217]FIG. 55 is a flowchart for describing the processing for regionspecification in detail.

[0218]FIG. 56 is a block diagram illustrating yet another configurationof the region specifying unit 103.

[0219]FIG. 57 is a block diagram describing the configuration of arobustification unit 361.

[0220]FIG. 58 is a diagram describing movement compensation of amovement compensation unit 381.

[0221]FIG. 59 is a diagram describing movement compensation of amovement compensation unit 381.

[0222]FIG. 60 is a flowchart describing the processing for regionspecification.

[0223]FIG. 61 is a flowchart describing details of processing forrobustification.

[0224]FIG. 62 is a block diagram illustrating an example of theconfiguration of a mixture ratio calculation unit 104.

[0225]FIG. 63 is a diagram illustrating an example of an ideal mixtureratio α.

[0226]FIG. 64 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0227]FIG. 65 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0228]FIG. 66 is a diagram describing approximation using correlation offoreground components.

[0229]FIG. 67 is a diagram describing the relation between C, N, and P.

[0230]FIG. 68 is a block diagram illustrating the configuration of anestimated mixture ratio processing unit 401.

[0231]FIG. 69 is a diagram illustrating an example of an estimatedmixture ratio.

[0232]FIG. 70 is a block diagram illustrating another configuration of amixture ratio calculation unit 104.

[0233]FIG. 71 is a flowchart describing processing for calculating amixture ratio.

[0234]FIG. 72 is a flowchart describing processing for computing anestimated mixture ratio.

[0235]FIG. 73 is a diagram describing a straight line approximating amixture ratio α.

[0236]FIG. 74 is a diagram describing a plane approximating a mixtureratio α.

[0237]FIG. 75 is a diagram describing how pixels in multiple framescorrespond at the time of calculating the mixture ratio α.

[0238]FIG. 76 is a block diagram illustrating another configuration ofthe mixture ratio estimation processing unit 401.

[0239]FIG. 77 is a diagram illustrating an example of an estimatedmixture ratio.

[0240]FIG. 78 is a flowchart explaining the processing for calculatingmixture ratio.

[0241]FIG. 79 is a flowchart describing the processing for mixture ratioestimation by way of a model corresponding to a covered backgroundregion.

[0242]FIG. 80 is a block diagram illustrating an example of theconfiguration of a foreground/background separation unit 105.

[0243]FIG. 81A is a diagram illustrating an input image, foregroundcomponent image, and background component image.

[0244]FIG. 81B is a model diagram corresponding to an input image,foreground component image, and background component image.

[0245]FIG. 82 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0246]FIG. 83 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0247]FIG. 84 is a model diagram wherein the pixel values are developedover the time direction, and periods corresponding to shutter time aredivided.

[0248]FIG. 85 is a block diagram illustrating an example of theconfiguration of the separation unit 601.

[0249]FIG. 86A is a diagram illustrating an example of a separatedforeground component image.

[0250]FIG. 86B is a diagram illustrating an example of a separatedbackground component image.

[0251]FIG. 87 is a flowchart describing the processing for separatingthe foreground and the background.

[0252]FIG. 88 is a block diagram illustrating an example of theconfiguration of a movement blurring removal unit 106.

[0253]FIG. 89 is a diagram describing increments of processing.

[0254]FIG. 90 is a model diagram wherein the pixel values of aforeground component image are developed over the time direction, andperiods corresponding to shutter time are divided.

[0255]FIG. 91 is a model diagram wherein the pixel values of foregroundcomponent image are developed over the time direction, and periodscorresponding to shutter time are divided.

[0256]FIG. 92 is a model diagram wherein the pixel values of foregroundcomponent image are developed over the time direction, and periodscorresponding to shutter time are divided.

[0257]FIG. 93 is a flowchart explaining processing for removing movementblurring contained in the foreground component image by the movementblurring removal unit 106.

[0258]FIG. 94 is a diagram illustrating a model of a backgroundcomponent image.

[0259]FIG. 95 is a diagram illustrating a model of a correctedbackground component image.

[0260]FIG. 96 is a block diagram illustrating the configuration of amovement-blurring-removed-image processing unit 108 for generatingcoefficient sets.

[0261]FIG. 97 is a diagram explaining the relation between a tutor imageand a student image.

[0262]FIG. 98 is a block diagram illustrating the configuration of alearning unit 1006.

[0263]FIG. 99A is a diagram explaining class classification processing.

[0264]FIG. 99B is a diagram explaining class classification processing.

[0265]FIG. 100A is a diagram explaining ADRC processing.

[0266]FIG. 100B is a diagram explaining ADRC processing.

[0267]FIG. 101 is a diagram explaining coefficient sets which themovement-blurring-removed-image processing unit 108 generates.

[0268]FIG. 102 is a flowchart explaining the learning processing forgenerating coefficient sets by the movement-blurring-removed-imageprocessing unit 108.

[0269]FIG. 103 is a flowchart explaining the processing for generatingcoefficient sets corresponding to the background component image.

[0270]FIG. 104 is a block diagram illustrating the configuration of themovement-blurring-removed-image processing unit 108 which executes classclassification adaptation processing and generates a higher resolutionimage in the spatial direction.

[0271]FIG. 105 is a diagram illustrating a model of a foregroundcomponent image wherein movement blurring has been removed.

[0272]FIG. 106 is a diagram illustrating a model of a foregroundcomponent image wherein movement blurring has been added.

[0273]FIG. 107 is a block diagram illustrating the configuration of amapping unit 1103.

[0274]FIG. 108 is a flowchart explaining the processing for creating animage with regard to the movement-blurring-removed-image processing unit108.

[0275]FIG. 109 is a flowchart explaining the processing of predictingimages corresponding to background component images.

[0276]FIG. 110 is a flowchart explaining the processing of images withthe image processing device according to the present invention.

[0277]FIG. 111 is a block diagram illustrating the configuration of themovement-blurring-removed-image processing unit 108 wherein edgeenhancing processing with difference effects is applied for each image.

[0278]FIG. 112 is a block diagram illustrating the configuration of anedge enhancing unit 1203.

[0279]FIG. 113A is a diagram describing the processing for edgeenhancement.

[0280]FIG. 113B is a diagram describing the processing for edgeenhancement.

[0281]FIG. 113C is a diagram describing the processing for edgeenhancement.

[0282]FIG. 114 is a diagram illustrating filter coefficients.

[0283]FIG. 115 is a diagram explaining operation of a high-pass filter1221.

[0284]FIG. 116 is a diagram illustrating filter coefficients.

[0285]FIG. 117 is a diagram explaining operation of a high-pass filter1221.

[0286]FIG. 118 is a block diagram illustrating the configuration of anedge enhancing unit 1203.

[0287]FIG. 119 is a diagram illustrating filter coefficients.

[0288]FIG. 120 is a diagram explaining operation of a filter 1241.

[0289]FIG. 121 is a diagram illustrating filter coefficients.

[0290]FIG. 122 is a diagram explaining operation of a filter 1241.

[0291]FIG. 123 is a diagram explaining the processing of themovement-blurring-removed-image processing unit 108.

[0292]FIG. 124 is a flowchart explaining the processing of edgeenhancement processing with the movement-blurring-removed-imageprocessing unit 108.

[0293]FIG. 125 is a block diagram illustrating the configuration of themovement-blurring-removed-image processing unit 108 which generatescoefficient sets.

[0294]FIG. 126 is a flowchart explaining the learning processing forgenerating coefficient sets used in class classification adaptationprocessing for removing noise.

[0295]FIG. 127 is a block diagram illustrating the configuration of themovement-blurring-removed-image processing unit 108.

[0296]FIG. 128 is a diagram explaining the processing of themovement-blurring-removed-image processing unit 108.

[0297]FIG. 129 is a flowchart explaining the processing of themovement-blurring-removed-image processing unit 108 having theconfiguration shown in FIG. 127.

[0298]FIG. 130 is a block diagram illustrating another configuration ofthe functions of the image processing device.

[0299]FIG. 131 is a block diagram illustrating an example of theconfiguration of a mixture ratio calculation unit 3001.

[0300]FIG. 132 is a block diagram illustrating an example of theconfiguration of a foreground/background separation unit 3002.

[0301]FIG. 133 is a block diagram illustrating another configuration ofthe functions of the image processing device.

[0302]FIG. 134 is a diagram explaining the processing of a separatedimage processing unit 4002.

[0303]FIG. 135 is a block diagram illustrating an example of theconfiguration of a foreground/background separation unit 4001.

[0304]FIG. 136 is a block diagram illustrating an example of theconfiguration of a separation unit 4101.

[0305]FIG. 137 is a block diagram illustrating the configuration of aseparated image processing unit 4002 for generating coefficient sets.

[0306]FIG. 138 is a block diagram illustrating the configuration of theseparated image processing unit 4002 which generates a higher resolutionimage in the spatial direction.

[0307]FIG. 139A is a diagram illustrating an example of an image in themixed region of a tutor image.

[0308]FIG. 139B is a diagram illustrating change in pixel values of animage in the mixed region of a tutor image.

[0309]FIG. 140A is a diagram illustrating an example of an image in amixed region, generated by conventional class classification adaptationprocessing.

[0310]FIG. 140B is a diagram illustrating change in the pixel values ofan image in a mixed region, generated by conventional classclassification adaptation processing.

[0311]FIG. 141A is a diagram illustrating an example of an image in amixed region, generated by the separated image processing unit 4002.

[0312]FIG. 141B is a diagram illustrating change in the pixel values ofan image in a mixed region, generated by the separated image processingunit 4002.

[0313]FIG. 142A is a diagram illustrating an example of an image in aforeground region of a tutor image.

[0314]FIG. 142B is a diagram illustrating change in the pixel values ofan image in a foreground region of a tutor image.

[0315]FIG. 143A is a diagram illustrating an example of an image in aforeground region, generated by conventional class classificationadaptation processing.

[0316]FIG. 143B is a diagram illustrating change in the pixel values ofan image in a foreground region, generated by conventional classclassification adaptation processing.

[0317]FIG. 144A is a diagram illustrating an example of an image in aforeground region, generated by the separated image processing unit4002.

[0318]FIG. 144B is a diagram illustrating change in the pixel values ofan image in a foreground region, generated by the separated imageprocessing unit 4002.

[0319]FIG. 145 is a flowchart explaining the processing of images withthe image processing device, the configuration of which is shown in FIG.133.

[0320]FIG. 146 is a flowchart explaining the processing of separatingthe foreground and background with the foreground/background separationunit 4001.

[0321]FIG. 147 is a flowchart explaining learning processing forgenerating coefficient sets by the separated image processing unit 4002.

[0322]FIG. 148 is a flowchart explaining the processing for creatingimages with the separated image processing unit 4002.

[0323]FIG. 149 is a block diagram illustrating the configuration of theseparated image processing unit 4002.

[0324]FIG. 150 is a diagram explaining the processing by the separatedimage processing unit 4002.

[0325]FIG. 151 is a flowchart explaining the processing of images withthe image processing device, the configuration of which is shown in FIG.133.

[0326]FIG. 152 is a flowchart explaining the processing of separatedimages with the separated image processing unit 4002.

[0327]FIG. 153 is a block diagram illustrating yet another configurationof the functions of the image processing device.

[0328]FIG. 154 is a block diagram illustrating an example of theconfiguration of a foreground/background separation unit 4601.

[0329]FIG. 155 is a block diagram illustrating another configuration ofthe functions of the image processing device.

[0330]FIG. 156 is a diagram explaining the processing of the regionprocessing unit 5001.

[0331]FIG. 157 is a block diagram illustrating the configuration of theregion processing unit 5001 which generates coefficient sets.

[0332]FIG. 158 is a block diagram illustrating the configuration of theregion processing unit 5001 which generates a higher resolution image inthe spatial direction.

[0333]FIG. 159A is a diagram illustrating an example of an image in themixed region of a tutor image.

[0334]FIG. 159B is a diagram illustrating change in pixel values of animage in the mixed region of a tutor image.

[0335]FIG. 160A is a diagram illustrating an example of an image in amixed region, generated by conventional class classification adaptationprocessing.

[0336]FIG. 160B is a diagram illustrating change in the pixel values ofan image in a mixed region, generated by conventional classclassification adaptation processing.

[0337]FIG. 161A is a diagram illustrating an example of an image in amixed region, generated by the region processing unit 5001.

[0338]FIG. 161B is a diagram illustrating change in the pixel values ofan image in a mixed region, generated by the region processing unit5001.

[0339]FIG. 162A is a diagram illustrating an example of an image in aforeground region of a tutor image.

[0340]FIG. 162B is a diagram illustrating change in the pixel values ofan image in a foreground region of a tutor image.

[0341]FIG. 163A is a diagram illustrating an example of an image in aforeground region, generated by conventional class classificationadaptation processing.

[0342]FIG. 163B is a diagram illustrating change in the pixel values ofan image in a foreground region, generated by conventional classclassification adaptation processing.

[0343]FIG. 164A is a diagram illustrating an example of an image in aforeground region, generated by the region processing unit 5001.

[0344]FIG. 164B is a diagram illustrating change in the pixel values ofan image in a foreground region, generated by the region processing unit5001.

[0345]FIG. 165 is a flowchart explaining the processing of images withthe image processing device, the configuration of which is shown in FIG.155.

[0346]FIG. 166 is a flowchart explaining the learning processing forgenerating coefficient sets with the region processing unit 5001.

[0347]FIG. 167 is a flowchart explaining the processing for creatingimages with the region processing unit 5001.

[0348]FIG. 168 is a block diagram illustrating the configuration of theregion processing unit 5001.

[0349]FIG. 169 is a diagram explaining the processing by the regionprocessing unit 5001.

[0350]FIG. 170 is a flowchart explaining the processing of images withthe image processing device, the configuration of which is shown in FIG.155.

[0351]FIG. 171 is a flowchart explaining the processing of edgeenhancement processing with the region processing unit 5001.

[0352]FIG. 172 is a flowchart explaining the processing of images withthe image processing device shown in FIG. 155.

[0353]FIG. 173 is a block diagram illustrating the configuration of theregion processing unit 5001 which generates coefficient sets.

[0354]FIG. 174 is a diagram explaining coefficient sets generated by theregion processing unit 5001, the configuration of which is shown in FIG.173.

[0355]FIG. 175 is a flowchart explaining the learning processing forgenerating coefficient sets.

[0356]FIG. 176 is a block diagram illustrating the configuration of theregion processing unit 5001 which generates images wherein the noise hasbeen removed.

[0357]FIG. 177 is a flowchart explaining the processing for creatingimages with the region processing unit 5001, the configuration of whichis shown in FIG. 176.

[0358]FIG. 178 is a block diagram illustrating a configuration of thefunctions of the image processing device.

[0359]FIG. 179 is a flowchart explaining the processing of images withthe image processing device according to the present invention.

[0360]FIG. 180 is a block diagram illustrating the configuration of aseparated image processing unit 7001 which generates coefficient sets.

[0361]FIG. 181 is a block diagram illustrating the configuration of alearning unit 7024.

[0362]FIG. 182 is a flowchart explaining the learning processing forgenerating coefficient sets with the separated image processing unit7001.

[0363]FIG. 183 is a flowchart for explaining the processing forgenerating coefficient sets corresponding to the background componentimage.

[0364]FIG. 184 is a block diagram illustrating the configuration of theseparated image processing unit 7001 which generates a higher resolutionimage in the spatial direction by applying the class classificationadaptation processing to the background component image, while alsoperforming linear interpolation of the foreground component image.

[0365]FIG. 185 is a block diagram illustrating the configuration of amapping unit 7302.

[0366]FIG. 186 is a diagram explaining the processing of the separatedimage processing unit 7001, the configuration of which is shown in FIG.184.

[0367]FIG. 187 is a flowchart explaining the processing of the separatedimage processing unit 7001 shown in FIG. 184.

[0368]FIG. 188 is a flowchart describing processing for predictingimages corresponding to the background component image.

[0369]FIG. 189 is a block diagram illustrating the configuration of theseparated image processing unit 7001 which applies edge enhancementprocessing to only the background component image.

[0370]FIG. 190 is a block diagram illustrating the configuration of anedge enhancing unit 7501.

[0371]FIG. 191 is a block diagram illustrating another configuration ofthe edge enhancing unit 7501.

[0372]FIG. 192 is a diagram explaining the processing of the separatedimage processing unit 7001, the configuration of which is shown in FIG.189.

[0373]FIG. 193 is a flowchart explaining the processing of the separatedimage processing unit 7001, the configuration of which is shown in FIG.189.

[0374]FIG. 194 is a block diagram illustrating another configuration ofthe functions of the image processing device.

BEST MODE FOR CARRYING OUT THE INVENTION

[0375]FIG. 10 is a block diagram which illustrates the configuration ofan embodiment of the image processing device according to the presentinvention. A CPU (Central Processing Unit) 71 performs various types ofprocessing following programs stored in ROM (Read Only Memory) 72, or astorage unit 78. RAM (Random Access Memory) 73 suitably stores programsfor the CPU 71 to execute, data, and so forth. These CPU 71, ROM 72, andRAM 73 are mutually connected via a bus 74.

[0376] The CPU 71 is also connected to an input/output interface 75 viathe bus 74. The input/output interface 75 is connected to an input unit76 such as a keyboard, mouse, microphone, or the like, and is connectedto an output unit 77 such as a display, speaker, or the like. The CPU 71performs various types of processing corresponding to instructions inputfrom the input unit 76. The CPU 71 then outputs images, audio, or thelike, which are obtained as a result of processing, to the output unit77.

[0377] The storage unit 78 connected to the input/output interface 75comprises a hard disk, for example, and stores programs for the CPU 71to execute and various types of data. A communication unit 79communicates with external devices via the Internet or other networks.In this case of the example, the communication unit 79 serves as anobtaining unit which obtains output from a sensor.

[0378] Also, an arrangement may be made wherein programs are obtainedvia the communication unit 79, and are stored in the storage unit 78.

[0379] A drive 80 connected to the input/output interface 75 drives amagnetic disk 91, optical disk 92, magneto-optical disk 93,semiconductor memory 94, or the like, in the event that those aremounted thereon, and obtains programs and data stored therein. Theobtained programs and data are transmitted to the storage unit 78 andstored therein, as necessary.

[0380]FIG. 11 is a block diagram which illustrates the configuration ofthe functions of the image processing device according to the presentinvention.

[0381] Note that whether each function of the image processing device isrealized by hardware or software does not matter. That is to say, eachblock diagram in the present Specification may be regarded as not only ahardware block diagram but also as a software function block diagram.

[0382] Note that movement blurring means distortion which is included inimages corresponding to moving objects, which occurs due to movement ofobjects which are objects of image-taking in the real world and due toimage-taking properties of the sensor.

[0383] In the present Specification, images corresponding to objectswhich are objects of image-taking in the real world are called imageobjects.

[0384] Input images provided to the image processing device are providedto an object extracting unit 101, a region specifying unit 103, amixture ratio calculation unit 104, and a foreground/backgroundseparation unit 105.

[0385] The object extracting unit 101 roughly extracts the image objectscorresponding to the foreground object contained in the input image, andsupplies the extracted image object to the movement detecting unit 102.The object extracting unit 101 roughly extracts the image objectcorresponding to the foreground object, for example, by detecting theoutlines of the image object corresponding to the foreground objectincluded in input image.

[0386] The object extracting unit 101 roughly extracts the image objectcorresponding to the background object contained in the input image, andsupplies the extracted image object to the movement detecting unit 102.The object extracting unit 101 roughly extracts the image objectcorresponding to the background object, by the difference between theinput image and the image object corresponding to the extractedforeground object, for example.

[0387] Also, for example, an arrangement may be made wherein the objectextracting unit 101 roughly extracts the image objects corresponding tothe foreground objects and the image objects corresponding to thebackground objects based upon the difference between the backgroundimages stored in background memory provided therein and the inputimages.

[0388] The movement detecting unit 102 calculates the movement vectorsof the image object corresponding to the roughly extracted foregroundobjects by techniques such as block matching, gradation, phasecorrelation, and pixel recursion, or the like, and provides thecalculated movement vectors and movement vector position information(information for specifying the pixel positions corresponding to themovement vectors) to the region specifying unit 103 and the movementblurring removal unit 106.

[0389] The movement vector output from the movement detecting unit 102includes information corresponding to a movement amount v.

[0390] Also, for example, an arrangement may be made wherein themovement detecting unit 102 outputs the movement vector for each imageobject to the movement blurring removal unit 106 along with the pixelposition information for specifying a pixel with regard to the imageobject.

[0391] The movement amount v is a value which represents the change ofposition of the image corresponding to the moving object in incrementsof pixel interval. For example, in the event that the object imagecorresponding to the foreground moves so as to be displayed at aposition four pixels away in the following frame with a given frame as areference, the movement amount v of the image of the objectcorresponding to the foreground is 4.

[0392] The region specifying unit 103 classifies each of input pixels ofan image into one of the foreground region, the background region, orthe mixed region, and supplies information which indicates which of theforeground region, the background region, or the mixed region, eachpixel belongs to, (which will be referred to as region informationhereafter), to the mixture ratio calculation unit 104, theforeground/background separation unit 105, and the movement blurringremoval unit 106. Details of the foreground region, the backgroundregion, or the mixed region, will be described later.

[0393] The mixture ratio calculation unit 104 calculates the mixtureratio corresponding to the pixels contained in the mixed region (whichwill be referred to as mixture ratio α hereafter) based upon the inputimage and the region information supplied from the region specifyingunit 103, and supplies the calculated mixed ratio to theforeground/background separation unit 105.

[0394] The mixture ratio α is a value which represents the ratio of theimage component corresponding to the background object (which will alsobe referred to as background component hereafter) with the pixel valueas indicated in Expression (3) described below.

[0395] The foreground/background separation unit 105 separates the inputimages into foreground component images which consist of only the imagecomponents corresponding to the foreground object (which will also bereferred to as the foreground component hereafter) and backgroundcomponent images which consist of only the background components basedupon the region information supplied from the region specifying unit 103and the mixture ratio α supplied from the mixture ratio calculation unit104, supplies the foreground component images to the movement blurringremoval unit 106, and supplies the background component image to acorrection unit 107.

[0396] The movement blurring removal unit 106 decides the increment ofprocessing, which indicates one or more pixels included in theforeground component images, based upon the movement amount v which isled from the movement vector, and the region information. An incrementof processing is the data which designates one group of the pixels whichare the object for adjustment processing for the movement blurringamount.

[0397] The movement blurring removal unit 106 removes movement blurringcontained in the foreground component image based upon the foregroundcomponent image supplied from the foreground/background separation unit105, the movement vector and the position information thereof suppliedfrom the movement detecting unit 102, and the processing increment, andoutputs the foreground component image which has been subjected toremoval of movement blurring, to a movement-blurring-removed-imageprocessing unit 108.

[0398] The correction unit 107 corrects the pixel value of a pixelcorresponding to the mixed region in the background component image. Thepixel value of a pixel corresponding to the mixed region in thebackground component image is calculated by subtracting the foregroundcomponent from the pixel value of a pixel in the mixed region prior toseparation. Accordingly, the pixel value of a pixel corresponding to themixed region in the background component image decreases correspondingto the mixture ratio α, as compared to the pixel value of a pixel in theadjacent background region.

[0399] The correction unit 107 corrects the decrease of the gaincorresponding to the mixture ratio α of the pixel value of a pixelcorresponding to the mixed region in the background component image, asdescribed above, and supplies the corrected background component imageto the movement-blurring-removed-image processing unit 108.

[0400] The movement-blurring-removed-image processing unit 108individually performs processing for the foreground component imagewhich has been subjected to removal of movement blurring and thecorrected background component image by classification adaptationprocessing.

[0401] For example, the movement-blurring-removed-image processing unit108 generates coefficients which are used in the classifying adaptationprocessing for generating even higher resolution image, for eachforeground component image which has been subjected to removal ofmovement blurring, and for each corrected background component image.

[0402] For example, the movement-blurring-removed-image processing unit108 creates an even higher resolution image by applying the classifyingadaptation processing to each foreground component image which has beensubjected to removal of movement blurring, and for each correctedbackground component image.

[0403] The input images supplied to the image processing device will nowbe described, referring to FIG. 12 through FIG. 27.

[0404]FIG. 12 is a diagram which describes image-taking by a sensor. Thesensor comprises a CCD video camera or the like, for example, includinga CCD (Charge-Coupled Device) area sensor which is a solid-stateimage-taking device. An object 111 corresponding to the foreground inthe real world moves between an object 112 corresponding to thebackground in the real world, and the sensor, for example, from the leftside to the right side horizontally in the drawing.

[0405] The sensor takes images of the object 111 corresponding to theforeground with the object 112 corresponding to the background. Thesensor outputs the taken images in increments of one frame. For example,the sensor outputs images of 30 frames per second. In this case, theexposure period of the sensor is {fraction (1/30)} seconds. The exposureperiod represents a period from the sensor beginning conversion of inputlight into electric charges, up to the end of conversion of input lightto electric charges. The exposure period will be also referred to as ashutter period hereafter.

[0406]FIG. 13 is a diagram which describes an arrangement of pixels. InFIG. 13, A through I denote individual pixels. These pixels are arrangedon a plane corresponding to the image. One detecting elementcorresponding to one pixel is disposed on the sensor. Upon the sensortaking images, one detecting element outputs pixel values correspondingto one pixel which makes up the image. For example, a position in the Xdirection of the detecting elements corresponds to a position in thehorizontal direction on the image, and a position in the Y direction ofthe detecting elements corresponds to a position in the verticaldirection on the image.

[0407] As shown in FIG. 14, for example, the detecting element of theCCD converts the input light into charges for a period corresponding tothe shutter period, and accumulates the converted charges. The quantityof charges is approximately proportional to the strength of the inputlight and the period during which the light is input. The detectingelement adds the charges converted from the input light to theaccumulated charges in the period corresponding to the shutter period.That is to say, the detecting element integrates the input light duringthe period corresponding to the shutter period, and accumulates theamount of charges corresponding to the integrated light. It can also besaid that the detecting element has integrating effects with regard totime.

[0408] The charges accumulated in the detecting element are convertedinto a voltage value by a circuit not shown in the drawings, which isfurther converted to pixel values such as digital data or the like, andis output. Accordingly, individual pixel values output from a sensorhave values projected in one-dimensional space, which is from a resultwherein a given portion having a spatial extension of the objectcorresponding to the foreground or the background, is integrated for theshutter period.

[0409] The image processing device extracts valid information buried inoutput signals due to such accumulation operations of the sensor, suchas the mixture ratio α, for example.

[0410]FIG. 15A and FIG. 15B are diagrams which describe the image whichis obtained by taking images of the object corresponding to the movingforeground and the object corresponding to the still background. FIG.15A illustrates the image which is obtained by taking image of theobject corresponding to the foreground with movement and the objectcorresponding to the still background. With the example shown in FIG.15A, the object corresponding to the foreground moves from the left tothe right horizontally in the drawing.

[0411]FIG. 15B is a model diagram wherein pixel values corresponding toone line of the image shown in FIG. 15A develop over the time direction.The horizontal direction in FIG. 15B corresponds to the spatialdirection X in FIG. 15A.

[0412] The pixel values of pixels in the background regions are made upof only the background components, i.e., the image componentscorresponding to the background objects. The pixel values of pixels inthe foreground regions are made up of only the foreground components,i.e., the image components corresponding to the foreground objects.

[0413] The pixel values of pixels in mixed regions are made up of thebackground components and the foreground components. Since the pixelvalues in the mixed region consists of the background components and theforeground components, the mixed region can also be said to be adistortion region. The mixed regions are further classified into coveredbackground regions and uncovered background regions.

[0414] The covered background region is a mixed region at a positioncorresponding to a leading portion in the progress direction of theforeground object with regard to the foreground region, and accordinglyis a region wherein the background components are covered by theforeground corresponding to elapsing of time.

[0415] Conversely, the uncovered background region is a mixed region ata position corresponding to a trailing portion in the progress directionof the foreground object with regard to the foreground region, andaccordingly is a region wherein the background components emergecorresponding to elapsing of time.

[0416] As described above, images including the foreground region,background region, covered background region, or uncovered backgroundregion, are input as input images to the region specifying unit 103, themixture ratio calculation unit 104, and the foreground/backgroundseparation unit 105.

[0417]FIG. 16 is a diagram which describes the background region,foreground region, mixed region, covered background region, anduncovered background region, as described above. In the event ofcorresponding to the images shown in FIG. 15A, the background region isthe still portion, the foreground region is the moving portion, thecovered background region of the mixed region is the portion whichchanges from the background to the foreground, and the uncoveredbackground region of the mixed region is the portion which changes fromthe foreground to the background.

[0418]FIG. 17 is a model diagram wherein pixel values of the pixelsarrayed adjacently in one line in the image that has been taken of theobjects corresponding to the still foregrounds and the objectscorresponding to the still backgrounds, develop over the time direction.For example, pixels arrayed in one line in a screen may be selected, aspixels adjacently arrayed in one line.

[0419] The pixel values F01 through F04 shown in FIG. 17 are pixelvalues of pixels corresponding to the still foreground object. The pixelvalues B01 through B04 shown in FIG. 17 are pixel values of pixelscorresponding to the still background object.

[0420] The vertical direction in FIG. 17 represents elapsing of timefrom the top to the bottom in the drawing. The position of the upperside of the rectangle in FIG. 17 corresponds to the time at which thesensor begins conversion of the input light into charges, and theposition of the lower side of the rectangle in FIG. 17 corresponds tothe time at which the sensor ends the conversion of the input light intocharges. That is to say, the distance from the upper side to the lowerside of the rectangle in FIG. 17 corresponds to the shutter period.

[0421] An arrangement wherein the shutter period equals the frameinterval will now be described below, by way of an example.

[0422] The horizontal direction in FIG. 17 corresponds to the spatialdirection X as described in FIG. 15A. More particularly, shown by way ofan example in FIG. 17, the distance from the left side of the rectangledenoted by “F01” to the right side of the rectangle denoted by “B04” inFIG. 17, is eight times long as the pixel pitch, that is to say,corresponds to the interval of eight continuous pixels.

[0423] In the event that the foreground objects and the backgroundobjects keep still, the light input to the sensor is not altered duringthe period corresponding to the shutter period.

[0424] Now, the period corresponding to the shutter period is dividedinto two or more periods of equal length. For example, in the event thatthe virtual dividing number is 4, the model diagram shown in FIG. 17 canbe represented by the model shown in FIG. 18. The virtual dividingnumber is set corresponding to the movement amount v or the like of theobject corresponding to the foreground within the shutter period. Forexample, corresponding to the movement amount v of 4, the virtualdividing number is 4, and the period corresponding to the shutter periodis divided into 4 periods.

[0425] The uppermost row in the drawing corresponds to the first of thedivided periods from the shutter being opened. The second row from thetop in the drawing corresponds to the second of the divided periods fromthe shutter being opened. The third row from the top in the drawingcorresponds to the third of the divided periods from the shutter beingopened. The fourth row from the top in the drawing corresponds to thefourth of the divided periods from the shutter being opened.

[0426] The divided shutter period corresponding to the movement amount vwill also be referred to as a shutter period/v hereafter.

[0427] In the event that the object corresponding to the foregroundkeeps still, the foreground component F01/v equals the value in whichthe pixel value F01 is divided by the virtual dividing number, since thelight input to the sensor is not altered. Similarly, in the event thatthe object corresponding to the foreground keeps still, the foregroundcomponent F02/v equals the value of the pixel value F02 being divided bythe virtual dividing number, the foreground component F03/v equals thevalue of the pixel value F03 being divided by the virtual dividingnumber, and the foreground component F04/v equals the value of the pixelvalue F04 being divided by the virtual dividing number.

[0428] In the event that the object corresponding to the backgroundkeeps still, the background component B01/v equals the value of thepixel value B01 being divided by the virtual dividing number, since thelight input to the sensor is not altered. Similarly, in the event thatthe object corresponding to the background keeps still, the backgroundcomponent B02/v equals the value of the pixel value B02 being divided bythe virtual dividing number, B03/v equals the value of the pixel valueB03 being divided by the virtual dividing number, and B04/v equals thevalue of the pixel value B04 being divided by the virtual dividingnumber.

[0429] That is to say, in the event that the object corresponding to theforeground keeps still, the foreground component F01/v corresponding tothe first shutter period/v from the shutter opening, the foregroundcomponent F01/v corresponding to the second shutter period/v from theshutter opening, the foreground component F01/v corresponding to thethird shutter period/v from the shutter opening, and the foregroundcomponent F01/v corresponding to the fourth shutter period/v from theshutter opening, are the same value, since the light corresponding tothe foreground object which is input to the sensor is not altered duringthe period corresponding to the shutter period. F02/v through F04/v havethe same relationship as F01/v.

[0430] In the event that the object corresponding to the backgroundkeeps still, the background component B01/v corresponding to the firstshutter period/v from the shutter opening, the background componentsB01/v corresponding to the second shutter period/v from the shutteropening, the background components B01/v corresponding to the thirdshutter period/v from the shutter opening, and the background componentsB01/v corresponding to the fourth shutter period/v from the shutteropening, are the same value, since the light corresponding to thebackground object which is input to the sensor is not altered during theperiod corresponding to the shutter period. B02/v through B04/v have thesame relationship.

[0431] A case will now be described wherein the object corresponding tothe foreground moves while the object corresponding to the backgroundkeeps still.

[0432]FIG. 19 is a model diagram wherein pixel values of the pixels onone line including the covered background region develop over the timedirection in the event that the object corresponding to the foregroundmoves towards the right side in the drawing. In FIG. 19, the movementamount v of the foreground is 4. Since one frame is a short period, anassumption may be made that the object corresponding to the foregroundis a rigid body, and moves at a constant velocity. In FIG. 19, theobject image corresponding to the foreground moves so as to be displayedat a position four pixels to the right in the following frame, with agiven frame as a reference.

[0433] In FIG. 19, the left-most pixel through the fourth pixel from theleft, belong to the foreground region. In FIG. 19, the fifth through theseventh pixels from the left belong to the covered background region ofthe mixed region. In FIG. 19, the right-most pixel belongs to thebackground region.

[0434] Since the object corresponding to the foreground moves so as tohide the object corresponding to the background with elapsing of time,the components contained in the pixel values of the pixels which belongto the covered background region change from the background componentsto the foreground components at a certain point of the periodcorresponding to the shutter period.

[0435] For example, the pixel value M shown with a heavy frame in FIG.19, is represented by Expression (1).

M=B02/v+B02/v+F07/v+F06/v  (1)

[0436] For example, since the fifth pixel from the left includes abackground component corresponding to one shutter period/v andforeground components corresponding to the three shutter period/vs, themixture ratio α of the fifth pixel from the left is one-quarter. Sincethe sixth pixel from the left includes background componentscorresponding to the two shutter period/vs and two foreground componentscorresponding to the two shutter period/vs, the mixture ratio α of thesixth pixel from the left is 1/2. Since the seventh pixel from the leftincludes background components corresponding to the three shutterperiod/vs and a foreground component corresponding to the one shutterperiod/v, the mixture ratio α of the seventh pixel from the left is 3/4.

[0437] Since an assumption may be made that the object corresponding tothe foreground is a rigid body and the foreground image moves at aconstant velocity so as to be displayed at a position four pixels to theright in the following frame, the foreground component F07/v of thefirst shutter period/v from the shutter opening of the fourth pixel fromthe left in FIG. 19, for example, equals the foreground componentcorresponding to the second shutter period/v from the shutter opening ofthe fifth pixel from the left in FIG. 19. Similarly, the foregroundcomponent F07/v equals the foreground component corresponding to thethird shutter period/v from the shutter opening of the sixth pixel fromthe left in FIG. 19, and the foreground component corresponding to thefourth shutter period/v from the shutter opening of the seventh pixelfrom the left in FIG. 19, respectively.

[0438] Since an assumption may be made that the object corresponding tothe foreground is a rigid body and that the foreground image moves at aconstant velocity so as to be displayed at a point four pixels to theright in the following frame, the foreground component F06/v of thefirst shutter period/v from the shutter opening of the third pixel fromthe left in FIG. 19, for example, equals the foreground componentcorresponding to the second shutter period/v from the shutter opening ofthe fourth pixel from the left in FIG. 19. Similarly, the foregroundcomponent F06/v equals the foreground component corresponding to thethird shutter period/v from the shutter opening of the fifth pixel fromthe left in FIG. 19, and the foreground component corresponding to thefourth shutter period/v from the shutter opening of the sixth pixel fromthe left in FIG. 19, respectively.

[0439] Since an assumption may be made that the object corresponding tothe foreground is a rigid body and the foreground image moves at aconstant velocity so as to be displayed at a position four pixels to theright in the following frame, the foreground component F05/v of thefirst shutter period/v from the shutter opening of the second pixel fromthe left in FIG. 19, for example, equals the foreground componentcorresponding to the second shutter period/v from the shutter opening ofthe third pixel from the left in FIG. 19. Similarly, the foregroundcomponent F05/v equals the foreground component corresponding to thethird shutter period/v from the shutter opening of the fourth pixel fromthe left in FIG. 19, and the foreground component corresponding to thefourth shutter period/v from the shutter opening of the fifth pixel fromthe left in FIG. 19, respectively.

[0440] Since an assumption may be made that the object corresponding tothe foreground is a rigid body and the foreground image moves at aconstant velocity so as to be displayed at a position four pixels to theright in the following frame, the foreground component F04/v of thefirst shutter period/v from the shutter opening of the left-most pixelin FIG. 19, for example, equals the foreground component correspondingto the second shutter period/v from the shutter opening of the secondpixel from the left in FIG. 19. Similarly, the foreground componentF04/v equals the foreground component corresponding to the third shutterperiod/v from the shutter opening of the third pixel from the left inFIG. 19, and the foreground component corresponding to the fourthshutter period/v from the shutter opening of the fourth pixel from theleft in FIG. 19, respectively.

[0441] As described above, the foreground region corresponding to themoving object includes movement blurring, so this can be said to be adistorted region.

[0442]FIG. 20 is a model diagram wherein the pixel values of the pixelson one line including the uncovered background region develop over thetime direction in the event that the foreground moves toward the rightside in the drawing. In FIG. 20, the movement amount v of the foregroundis 4. Since one frame is a short time, an assumption may be made thatthe object corresponding to the foreground is a rigid body, and moves ata constant velocity. In FIG. 20, the object image corresponding to theforeground moves to the right side by four pixels in the following framewith a given frame as a reference.

[0443] In FIG. 20, the left-most pixel through the fourth pixel from theleft, belong to the background region. In FIG. 20, the fifth through theseventh pixels from the left belong to the mixed region of the uncoveredbackground. In FIG. 20, the right-most pixel belongs to the foregroundregion.

[0444] Since the object corresponding to the foreground which has hiddenthe object corresponding to the background moves so as to be removedfrom the front of the object corresponding to the background withelapsing of time, the components included in the pixel values of thepixels which belong to the uncovered background region change from theforeground components to the background components at a certain point inthe period corresponding to the shutter period.

[0445] For example, the pixel value M′ indicated with a heavy frame inFIG. 20, is represented by Expression (2).

M′=F02/v+F01/v+B26/v+B26/v  (2)

[0446] For example, since the fifth pixel from the left includes thebackground components corresponding to the three shutter period/vs, andthe foreground component corresponding to the one shutter period/v, themixture ratio α of the fifth pixel from the left is 3/4. Since the sixthpixel from the left includes the background components corresponding tothe two shutter period/vs and the foreground components corresponding tothe two shutter period/vs, the mixture ratio α of the sixth pixel fromthe left is 1/2. Since the seventh pixel from the left includes thebackground component corresponding to the one shutter period/v and theforeground components corresponding to the three shutter period/vs, themixture ratio α of the seventh pixel from the left is one-quarter.

[0447] Further generalizing Expression (1) and Expression (2), the pixelvalue M is represented by Expression (3). $\begin{matrix}{M = {{\alpha \cdot B} + {\sum\limits_{i}F_{i/v}}}} & (3)\end{matrix}$

[0448] Here, α denotes the mixture ratio. B denotes the pixel value ofthe background, and Fi/v denotes the foreground component.

[0449] Since an assumption may be made that the object corresponding tothe foreground is a rigid body and moves at a constant velocity, and themovement amount v is 4, for example, the foreground component F01/v ofthe first shutter period/v from the shutter opening of the fifth pixelfrom the left in FIG. 20 equals the foreground component correspondingto the second shutter period/v from the shutter opening of the sixthpixel from the left in FIG. 20. Similarly, F01/v equals the foregroundcomponent corresponding to the third shutter period/v from the shutteropening of the seventh pixel from the left in FIG. 20, and theforeground component corresponding to the fourth shutter period/v fromthe shutter opening of the eighth pixel from the left in FIG. 20,respectively.

[0450] Since an assumption may be made that the object corresponding tothe foreground is a rigid body and moves at a constant velocity, and thevirtual dividing number is 4, the foreground component F02/v of thefirst shutter period/v from the shutter opening of the sixth pixel fromthe left in FIG. 20, for example, equals the foreground componentcorresponding to the second shutter period/v from the shutter opening ofthe seventh pixel from the left in FIG. 20. Similarly, the foregroundcomponent F02/v equals the foreground component corresponding to thethird shutter period/v from the shutter opening of the eighth pixel fromthe left in FIG. 20.

[0451] Since an assumption may be made that the object corresponding tothe foreground is an rigid body and moves at a constant velocity, andmovement amount v is 4, the foreground component F03/v of the firstshutter period/v from the shutter opening of the seventh pixel from theleft in FIG. 20, for example, equals the foreground componentcorresponding to the second shutter period/v from the shutter opening ofthe eighth pixel from the left in FIG. 20.

[0452] While a description has been made in the description of FIG. 18through FIG. 20 wherein the virtual dividing number is 4, the virtualdividing number corresponds to the movement amount v. The movementamount v generally corresponds to the movement velocity of the objectcorresponding to the foreground. For example, in the event that theobject corresponding to the foreground moves so as to be displayed at aposition four pixels to the right in the following frame with a givenframe as a reference, the movement amount v is 4. The virtual dividingnumber is 4 corresponding to the movement amount v. Similarly, forexample, in the event that the object corresponding to the foregroundmoves so as to be displayed at a position six pixels to the left in thefollowing frame with a given frame as a reference, the movement amount vis 6, and the virtual dividing number is 6.

[0453]FIG. 21 and FIG. 22 illustrate the relationship between theforeground region, the background region, and the mixed region whichconsists of the covered background region or the uncovered backgroundregion, and the foreground components and the background components,corresponding to the divided shutter period.

[0454]FIG. 21 illustrates an example wherein the pixels of theforeground region, the background region, and the mixed region, areextracted from the image including the foreground corresponding to theobject which moves in front of the still background. In an example shownin FIG. 21, the object corresponding to the foreground denoted byreference character A moves horizontally on the screen.

[0455] The frame #n+1 is the frame following the frame #n, and the frame#n+2 is the frame following the frame #n+1.

[0456]FIG. 22 illustrates a model wherein the pixels of the foregroundregion, the background region, and the mixed region are extracted fromone of frame #n through frame #n+2, and the pixel values of theextracted pixels are developed over the time direction, with themovement amount v at 4.

[0457] Since the object corresponding to the foreground moves, the pixelvalues of the foreground region consist of four different foregroundcomponents corresponding to the period of shutter period/v. For example,the left-most pixel of the pixels of the foreground region shown in FIG.22 consists of F01/v, F02/v, F03/v, and F04/v. That is to say, thepixels of the foreground region include movement blurring.

[0458] Since the object corresponding to the background keeps still, thelight corresponding to the background input to the sensor is not alteredduring the period corresponding to the shutter period. In this case, thepixel values of the background region do not contain movement blurring.

[0459] The pixel value of the pixel which belongs to the mixed regionmade up of the covered background region or the uncovered backgroundregion consists of the foreground components and the backgroundcomponents.

[0460] Next, a model will be described wherein, in the event that theimage corresponding to the object moves, the pixel values of the pixelswhich are arrayed adjacently in a single line on multiple frames, and atthe same position in the frames, develop over the time direction. Forexample, in the event that the image corresponding to the object moveshorizontally on the screen, the pixels arrayed in a single line can beselected as pixels arrayed adjacently in a single line.

[0461]FIG. 23 is a model diagram wherein the pixel values of pixelsarrayed adjacently in a single line on three frames of images which aretaken of the object corresponding to the still background, and are atthe same position in the frames, develop over the time direction. Theframe #n is the frame following the frame #n−1, and the frame #n+1 isthe frame following the frame #n. Other frames are denoted in the sameway.

[0462] The pixel values of the B01 through B12 shown in FIG. 23 are thepixel values of the pixels corresponding to the object of the stillbackground. Since the object corresponding to the background keepsstill, the pixel values of the corresponding pixels do not change in theframe #n−1 through the frame #n+1. For example, the pixels in the frame#n and the pixels in the frame #n+1 at the position corresponding to thepixel having a pixel value B05 in the frame #n−1, have a pixel valueB05, respectively.

[0463]FIG. 24 is a model diagram wherein the pixel values of pixelsarrayed adjacently in a single line on three frames of images taken ofthe object corresponding to the foreground which moves to the right sidein the drawing with the object corresponding to the still background,and at the same position in the frames, develop over the time direction.The models shown in FIG. 24 includes the covered background region.

[0464] Since an assumption may be made in FIG. 24 that the objectcorresponding to the foreground is a rigid body and moves at a constantvelocity, and the foreground image moves so as to be displayed at aposition four pixels to the right side in the following frame, theforeground movement amount v is 4, and the virtual dividing number is 4.

[0465] For example, the foreground component of the first shutterperiod/v from the shutter opening of the left-most pixel of the frame#n−1 in FIG. 24 is F12/v, the foreground component of the second shutterperiod/v from the shutter opening of the second pixel from the left inFIG. 24 is also F12/v. The foreground component of the third shutterperiod/v from the shutter opening of the third pixel from the left inFIG. 24, and the foreground component of the fourth shutter period/vfrom the shutter opening of the fourth pixel from the left in FIG. 24,are F12/v.

[0466] The foreground component of the second shutter period/v from theshutter opening of the left-most pixel in the frame #n−1 in FIG. 24 isF11/v, and the foreground component of the third shutter period/v fromthe shutter opening of the second pixel from the left in FIG. 24 is alsoF11/v. The foreground component of the fourth shutter period/v from theshutter opening of the third pixel from the left in FIG. 24 is F11/v.

[0467] The foreground component of the third shutter period/v from theshutter opening of the left-most pixel in the frame #n−1 in FIG. 24 isF10/v, and the foreground component of the fourth shutter period/v fromthe shutter opening of the second pixel from the left in FIG. 24 is alsoF10/v. The foreground component of the fourth shutter period/v from theshutter opening of the left-most pixel in the frame #n−1 in FIG. 24 isF09/v.

[0468] Since the object corresponding to the background keeps still, thebackground component of the first shutter period/v from the shutteropening of the second pixel from the left in the frame #n−1 in FIG. 24is B01/v. The background components of the first and second shutterperiod/vs from the shutter opening of the third pixel from the left inthe frame #n−1 in FIG. 24 are B02/v. The background components of thefirst through third shutter period/vs from the shutter opening of thefourth pixel from the left in the frame #n−1 in FIG. 24 are B03/v.

[0469] In the frame #n−1 in FIG. 24, the left-most pixel belongs to theforeground region, and the second through fourth pixels from the leftbelong to the mixed region of the covered background region.

[0470] The fifth through twelfth pixels from the left in the frame #n−1in FIG. 24 belong to the background region, and the pixel values thereofare B04 through B11, respectively.

[0471] The first through fifth pixels from the left in the frame #n inFIG. 24 belong to the foreground region. The foreground component of theshutter period/v in the foreground region in the frame #n, is one ofF05/v through F12/v.

[0472] Since an assumption may be made that the object corresponding tothe foreground is a rigid body and moves at a constant velocity, and theforeground image moves so as to be displayed at a position four pixelsto the right side in the following frame, the foreground component ofthe first shutter period/v from the shutter opening of the fifth pixelfrom the left in the frame #n in FIG. 24 is F12/v, the foregroundcomponent of the second shutter period/v from the shutter opening of thesixth pixel from the left in FIG. 24 is also F12/v. The foregroundcomponent of the third shutter period/v from the shutter opening of theseventh pixel from the left in FIG. 24, and the foreground component ofthe fourth shutter period/v from the shutter opening of the eighth pixelfrom the left in FIG. 24, are F12/v.

[0473] The foreground component of the second shutter period/v from theshutter opening of the fifth pixel from the left in the frame #n in FIG.24 is F11/v, and the foreground component of the third shutter period/vfrom the shutter opening of the sixth pixel from the left in FIG. 24 isalso F11/v. The foreground component of the fourth shutter period/v fromthe shutter opening of the seventh pixel from the left in FIG. 24 isF11/v.

[0474] The foreground component of the third shutter period/v from theshutter opening of the fifth pixel from the left in the frame #n in FIG.24 is F10/v, and the foreground component of the fourth shutter period/vfrom the shutter opening of the sixth pixel from the left in FIG. 24 isalso F10/v. The foreground component of the fourth shutter period/v fromthe shutter opening of the fifth pixel from the left in the frame #n inFIG. 24 is F09/v.

[0475] Since the object corresponding to the background keeps still, thebackground component of the first shutter period/v from the shutteropening of the sixth pixel from the left in the frame #n in FIG. 24 isB05/v. The background components of the first and second shutterperiod/vs from the shutter opening of the seventh pixel from the left inthe frame #n in FIG. 24 are B06/v. The background components of thefirst through third shutter period/vs from the shutter opening of theeighth pixel from the left in the frame #n in FIG. 24 are B07/v.

[0476] In the frame #n in FIG. 24, the sixth through eighth pixels fromthe left belong to the mixed region of the covered background region.

[0477] The ninth through twelfth pixels from the left in the frame #n inFIG. 24 belong to the background region, and the pixel values are B08through B11, respectively.

[0478] The first through ninth pixels from the left in the frame #n+1 inFIG. 24 belong to the foreground region. The foreground component of theshutter period/v in the foreground region in the frame #n+1 is one ofF01/v through F12/v.

[0479] Since an assumption may be made that the object corresponding tothe foreground is a rigid body and moves at a constant velocity, and theforeground image moves so as to be displayed at a position four pixelsto the right side in the following frame, the foreground component ofthe first shutter period/v from the shutter opening of the ninth pixelfrom the left in the frame #n+1 in FIG. 24 is F12/v, and the foregroundcomponent of the second shutter period/v from the shutter opening of thetenth pixel from the left in FIG. 24 is also F12/v. The foregroundcomponent of the third shutter period/v from the shutter opening of theeleventh pixel from the left in FIG. 24, and the foreground component ofthe fourth shutter period/v from the shutter opening of the twelfthpixel from the left in FIG. 24, are F12/v.

[0480] The foreground component of the second shutter period/v from theshutter opening of the ninth pixel from the left in the frame #n+1 inFIG. 24 is F11/v, and the foreground component of the third shutterperiod/v from the shutter opening of the tenth pixel from the left inFIG. 24 is also F11/v. The foreground component of the fourth shutterperiod/v from the shutter opening of the eleventh pixel from the left inFIG. 24 is F11/v.

[0481] The foreground component of the third shutter period/v from theshutter opening of the ninth pixel from the left in the frame #n+1 inFIG. 24 is F10/v, and the foreground component of the fourth shutterperiod/v from the shutter opening of the tenth pixel from the left inFIG. 24 is also F10/v. The foreground component of the fourth shutterperiod/v from the shutter opening of the ninth pixel from the left inthe frame #n+1 in FIG. 24 is F09/v.

[0482] Since the object corresponding to the background keeps still, thebackground component of the first shutter period/v from the shutteropening of the tenth pixel from the left in the frame #n+1 in FIG. 24 isB09/v. The background components of the first and second shutterperiod/vs from the shutter opening of the eleventh pixel from the leftin the frame #n+1 in FIG. 24 are B10/v. The background components of thefirst through third shutter period/vs from the shutter opening of thetwelfth pixel from the left in the frame #n+1 in FIG. 24 are B11/v.

[0483] In the frame #n+1 in FIG. 24, the tenth through twelfth pixelsfrom the left side correspond to the mixed region which is the coveredbackground region.

[0484]FIG. 25 is a model diagram wherein the foreground components areextracted from the pixel values illustrated in FIG. 24.

[0485]FIG. 26 is a model diagram wherein the pixel values of the pixelsadjacently arrayed in a row in three frames of the images which aretaken of the foreground corresponding to the object which moves to theright side in the drawing with the still background, and are at the sameposition in the frames, develop over the time direction. In FIG. 26, themodel diagram includes the uncovered background region.

[0486] In FIG. 26, an assumption may be made that the objectcorresponding to the foreground is a rigid body, and moves at a constantvelocity. Since the object corresponding to the foreground moves so asto be displayed at a position four pixels to the right side in thefollowing frame, the movement amount v is 4.

[0487] For example, the foreground component of the first shutterperiod/v from the shutter opening of the left-most pixel in the frame#n−1 in FIG. 26 is F13/v, and the foreground component of the secondshutter period/v from the shutter opening of the second pixel from theleft in FIG. 26 is also F13/v. The foreground component of the thirdshutter period/v from the shutter opening of the third pixel from theleft in FIG. 26, and the foreground component of the fourth shutterperiod/v from the shutter opening of the fourth pixel from the left inFIG. 26, are F13/v.

[0488] The foreground component of the first shutter period/v from theshutter opening of the second pixel from the left in the frame #n−1 inFIG. 26 is F14/v, and the foreground component of the second shutterperiod/v from the shutter opening of the third pixel from the left inFIG. 26 is also F14/v. The foreground component of the first shutterperiod/v from the shutter opening of the third pixel from the left inFIG. 26 is F15/v.

[0489] Since the object corresponding to the background keeps still, thebackground component of the second through fourth shutter period/vs fromthe shutter opening of the left-most pixel in the frame #n−1 in FIG. 26is B25/v. The background components of the third and fourth shutterperiod/vs from the shutter opening of the second pixel from the left inthe frame #n−1 in FIG. 26 are B26/v. The background component of thefourth shutter period/v from the shutter opening of the third pixel fromthe left in the frame #n−1 in FIG. 26 is B27/v.

[0490] In the frame #n−1 in FIG. 26, the left-most pixel through thethird pixel belong to a mixed region of the uncovered background region.

[0491] The fourth through twelfth pixels from the left in the frame #n−1in FIG. 26 belong to the foreground region. The foreground component inthe frame is one of F13/v through F24/v.

[0492] The left-most pixel through the fourth pixel from the left in theframe #n in FIG. 26 belong to the background region, and the pixelvalues are B25 through B28, respectively.

[0493] Since an assumption may be made that the object corresponding tothe foreground is a rigid body and moves at a constant velocity, and theforeground image moves so as to be displayed at a position four pixelsto the right side in the following frame, the foreground component ofthe first shutter period/v from the shutter opening of the fifth pixelfrom the left in the frame #n in FIG. 26 is F13/v, and the foregroundcomponent of the second shutter period/v from the shutter opening of thesixth pixel from the left in FIG. 26 is also F13/v. The foregroundcomponent of the third shutter period/v from the shutter opening of theseventh pixel from the left in FIG. 26, and the foreground component ofthe fourth shutter period/v from the shutter opening of the eighth pixelfrom the left in FIG. 26, are F13/v.

[0494] The foreground component of the first shutter period/v from theshutter opening of the sixth pixel from the left in the frame #n in FIG.26 is F14/v, and the foreground component of the second shutter period/vfrom the shutter opening of the seventh pixel from the left in FIG. 26is also F14/v. The foreground component of the first shutter period/vfrom the shutter opening of the eighth pixel from the left in FIG. 26 isF15/v.

[0495] Since the object corresponding to the background keeps still, thebackground components of the second through fourth shutter period/vsfrom the shutter opening of the fifth pixel from the left in the frame#n in FIG. 26 are B29/v. The background components of the third andfourth shutter period/vs from the shutter opening of the sixth pixelfrom the left in the frame #n in FIG. 26 are B30/v. The backgroundcomponent of the fourth shutter period/v from the shutter opening of theseventh pixel from the left in the frame #n in FIG. 26 is B31/v.

[0496] In the frame #n in FIG. 26, the fifth through seventh pixels fromthe left belong to the mixed region of the uncovered background region.

[0497] The eighth through twelfth pixels from the left in the frame #nin FIG. 26 belong to the foreground region. The value corresponding tothe period of the shutter period/v in the foreground region in the frame#n is one of F13/v through F20/v.

[0498] The left-most pixel through the eighth pixel from the left in theframe #n+1 in FIG. 26, belong to the background region, and the pixelvalues thereof are B25 through B32, respectively.

[0499] Since an assumption may be made that the object corresponding tothe foreground is a rigid body and moves at a constant velocity, and theforeground image moves so as to be displayed at a position four pixelsto the right side in the following frame, the foreground component ofthe first shutter period/v from the shutter opening of the ninth pixelfrom the left in the frame #n+1 in FIG. 26 is F13/v, and the foregroundcomponent of the second shutter period/v from the shutter opening of thetenth pixel from the left in FIG. 26 is also F13/v. The foregroundcomponent of the third shutter period/v from the shutter opening of theeleventh pixel from the left in FIG. 26, and the foreground component ofthe fourth shutter period/v from the shutter opening of the twelfthpixel from the left in FIG. 26, are F13/v.

[0500] The foreground component of the first shutter period/v from theshutter opening of the tenth pixel from the left in the frame #n+1 inFIG. 26 is F14/v, and the foreground component of the second shutterperiod/v from the shutter opening of the eleventh pixel from the left inFIG. 26 is also F14/v. The foreground component of the first shutterperiod/v from the shutter opening of the twelfth pixel from the left inFIG. 26 is F15/v.

[0501] Since the object corresponding to the background keeps still, thebackground components of the second through fourth shutter period/vsfrom the shutter opening of the ninth pixel from the left in the frame#n+1 in FIG. 26 are B33/v. The background components of the third andfourth shutter period/vs from the shutter opening of the tenth pixelfrom the left in the frame #n+1 in FIG. 26 are B34/v. The backgroundcomponent of the fourth shutter period/v from the shutter opening of theeleventh pixel from the left in the frame #n+1 in FIG. 26 is B35/v.

[0502] In the frame #n+1 in FIG. 26, the ninth through eleventh pixelsfrom the left belong to the mixed region of the uncovered backgroundregion.

[0503] The twelfth pixel from the left in the frame #n+1 in FIG. 26belongs to the foreground region. The foreground component of theshutter period/v in the foreground region in the frame #n+1 is one ofF13/v through F16/v.

[0504]FIG. 27 is a model diagram of the image wherein the foregroundcomponents are extracted from the pixel values shown in FIG. 26.

[0505]FIG. 28 is a diagram which illustrates the correspondence of theimage divided into pixels each of which belongs to the foregroundregion, background region, covered background region, or uncoveredbackground region, to a model diagram wherein the pixel values of pixelsdevelop over the time direction.

[0506] As shown in FIG. 28, the region specifying unit 103 specifies theforeground region, background region, covered background region, anduncovered background region, of the input image.

[0507]FIG. 29 is a diagram which illustrates the correspondence of theinput image divided into the image of the foreground region, the imageof the background region, the foreground component image of the coveredbackground region, the background components of the covered backgroundregion, the foreground components of the uncovered background region,and the background components of the uncovered background region, to amodel diagram wherein the pixel values of pixels develop over the timedirection.

[0508] As shown in FIG. 29, the input image is classified into theforeground region, background region, covered background region, anduncovered background region, by the region specifying unit 103. Theinput image is separated into the image of the foreground region, theforeground components of the covered background region, and theforeground component image made up of the foreground components of theuncovered background region, and the image of the background region, thebackground components of the covered background region, and thebackground component image made up of the background components of theuncovered background region, based upon the foreground region,background region, covered background region, and uncovered backgroundregion, specified by foreground/background separation unit 105, and themixture ratio α detected by the mixture ratio calculation unit 104.

[0509] The separated foreground component image and background componentimage are processed for each image.

[0510] An arrangement may be made wherein the foreground/backgroundseparation unit 105 separates the input image into the image of theforeground region, image of the background region, foreground componentimage of the covered background region, background component image ofthe covered background region, foreground component image of theuncovered background region, and background component image of theuncovered background region, based upon the region information and themixture ratio α.

[0511]FIG. 30 is a diagram which illustrates an example of the imagedivided into the foreground region, the background region, and the mixedregion. The region specifying unit 103 specifies the foreground region,background region, and mixed region, of the input image. The imageprocessing device can divide the input image into the image of theforeground region, image of the background region, and image of themixed region, based upon the region information indicating theforeground region, background region, and mixed region.

[0512] As shown in FIG. 31, the foreground/background separation unit105 separates the image of the mixed region into the foregroundcomponent image of the mixed region and the background component imageof the mixed region, based upon the region information supplied from theregion specifying unit 103 and the mixture ratio α supplied from themixture ratio calculation unit 104.

[0513] As shown in FIG. 32, the separated background component image issubjected to correction with regard to the pixel values of the mixedregion, and the separated foreground component image is subjected toremoval of movement blurring.

[0514] As shown in FIG. 33, the input image is divided into regions, andseparated into the foreground components and the background components.The separated input image is synthesized into the foreground componentimage and the background component image.

[0515] The movement blurring contained in the foreground component imageis removed. The background component image is corrected for the pixelvalues corresponding to the mixed region.

[0516] The foreground component image which has been subjected toremoval of movement blurring, and the corrected background componentimage is individually performed processing.

[0517]FIG. 34 is a flowchart which describes the processing of the imageof the image processing device according to the present invention.

[0518] In Step S101, the region specifying unit 103 specifies theforeground region, background region, covered background region, anduncovered background region of the input image, based upon the movementvector and the position information thereof supplied from the movementdetecting unit 102 and the input image. Details of the processing forregion specifying will be described later.

[0519] In Step S102, the mixture ratio calculation unit 104 calculatesthe mixture ratio α based upon the region information supplied from theregion specifying unit 103 and the input image. Details of theprocessing of the mixture ratio calculation unit 104 calculating themixture ratio α will be described later.

[0520] In Step S103, the foreground/background separation unit 105separates the input image into the foreground component image made up ofthe foreground components and the background component image made up ofthe background components, based upon the region information suppliedfrom the region specifying unit 103 and the mixture ratio α suppliedfrom the mixture ratio calculation unit 104. Details of processing ofseparation of the image by the foreground/background separation unit 105will be described later.

[0521] In Step S104, the movement blurring removal unit 106 removesmovement blurring from the foreground component image supplied from theforeground/background separation unit 105, based upon the movementvector and the position information thereof supplied from the movementdetecting unit 102 and the region information supplied from the regionspecifying unit 103.

[0522] In Step S105, the correction unit 107 corrects the pixel valuescorresponding to the mixed region of the background component imagesupplied from the foreground/background separation unit 105.

[0523] In Step S106, the movement-blurring-removed image processing unit108 performs processing of the image for each foreground component imagewhich has been subjected to removal of movement blurring and eachbackground component image which has been corrected, and processingends. Details of the image processing which themovement-blurring-removed image processing unit 108 performs, will bedescribed later.

[0524] As described above, the image processing device according to thepresent invention separates the input image into the foregroundcomponent image and the background component image, removes movementblurring from the foreground component image, and performs processingfor each foreground component image which has been subjected to removalof movement blurring, and each background component image.

[0525] Description with regard to each configuration of the regionspecifying unit 103, the mixture ratio calculation unit 104, theforeground/background separation unit 105, the movement blurring removalunit 106, and the movement-blurring-removed image processing unit 108,will be made below.

[0526]FIG. 35 is a block diagram which illustrates an example of theconfiguration of the region specifying unit 103. The region specifyingunit 103, of which the structure is shown in FIG. 35, does not use themovement vectors. Frame memory 201 stores the input images in incrementsof one frame. In the event that the object of the processing is theframe #n, the frame memory 201 stores the frame #n−2 which is two framesprevious from the frame #n, the frame #n−1 which is one frame previousfrom the frame #n, the frame #n, the frame #n+1 which is one framefollowing the frame #n, and the frame #n+2 which is two frames followingthe frame #n.

[0527] A still/motion judgment unit 202-1 reads out the pixel value ofthe pixel in the frame #n+2, which is at the same position as theposition of the pixel on the image, which is the object of specifyingthe region in the frame #n, and the pixel value of the pixel in theframe #n+1, which is at the same position as the position of the pixelon the image, which is the object of specifying the region of the frame#n, from the frame memory 201, and calculates the absolute value of thedifference between the read out pixel values. The still/motion judgmentunit 202-1 judges whether or not the absolute value of the differencebetween the pixel value in the frame #n+2 and the pixel value in theframe #n+1 is greater than the predetermined threshold value Th, and inthe event that judgment is made that the absolute value of thedifference is greater than the threshold value Th, the still/motionjudgment unit 202-1 supplies the still/motion judgment, indicatingmotion, to a region judgment unit 203-1. In the event that judgment ismade that the absolute value of the difference between the pixel valueof the pixel in the frame #n+2 and the pixel value of the pixel in theframe #n+1 is equal to or less than the threshold value Th, thestill/motion judgment unit 202-1 supplies the still/motion judgment,indicating “still”, to the region judgment unit 203-1.

[0528] A still/motion judgment unit 202-2 reads out the pixel value ofthe pixel in the frame #n+1, which is at the same position as theposition of the pixel on the image, which is the object of specifyingthe region in the frame #n, and the pixel value of pixel which is theobject in the frame #n from the frame memory 201, and calculates theabsolute value of the difference between the pixel values. Thestill/motion judgment unit 202-2 judges whether or not the absolutevalue of the difference between the pixel value in the frame #n+1 andthe pixel value in the frame #n is greater than the predeterminedthreshold value Th, and in the event that judgment is made that theabsolute value of the difference between the pixel values is greaterthan the threshold value Th, the still/motion judgment indicating motionis supplied to the region judgment unit 203-1 and the region judgmentunit 203-2. In the event that judgment is made that the absolute valueof the difference between the pixel value of the pixel in the frame #n+1and the pixel value of the pixel in the frame #n is equal to or smallerthan the threshold value Th, the still/motion judgment unit 202-2supplies the still/motion judgment, indicating “still”, to the regionjudgment unit 203-1 and the region judgment unit 203-2.

[0529] The still/motion judgment unit 202-3 reads out the pixel value ofthe pixel, which is the object of specifying the region in the frame #n,and the pixel value of the pixel in the frame #n−1, which is at the sameposition as the position on the image of the pixel, which is the objectof specifying the region in the frame #n, from the frame memory 201, andcalculates the absolute value of the difference between the pixelvalues. The still/motion judgment unit 202-3 judges whether or not theabsolute value of the difference between the pixel value in the frame #nand the pixel value in the frame #n−1 is greater than the predeterminedvalue Th, and in the event that judgment is made that the absolute valueof the difference between the pixel values is greater than the thresholdvalue Th, the still/motion judgment indicating motion is supplied to theregion judgment unit 203-2 and the region judgment unit 203-3. In theevent that judgment is made that the absolute value of the differencebetween the pixel value of the pixel in the frame #n and the pixel valueof the pixel in the frame #n−1 is equal to or smaller than the thresholdvalue Th, the still/motion judgment unit 202-3 supplies the still/motionjudgment indicating “still” to the region judgment unit 203-2 and theregion judgment unit 203-3.

[0530] The still/motion judgment unit 202-4 reads out the pixel value ofthe pixel in the frame #n−1 at the same position as the position of thepixel on the image, which is the object of specifying the region in theframe #n, and the pixel value of the pixel in the frame #n−2 at the sameposition as the position of the pixel on the image, which is the objectof specifying the region in the frame #n, from the frame memory 201, andcalculates the absolute value of the difference between the pixelvalues. The still/motion judgment unit 202-4 judges whether or not theabsolute value of the difference between the pixel value in the frame#n−1 and the pixel value in the frame #n−2 is greater than thepredetermined threshold value Th, and in the event that judgment is madethat the absolute value of the difference between the pixel values isgreater than the threshold value Th, the still/motion judgmentindicating motion is supplied to the region judgment unit 203-3. In theevent that judgment is made that the absolute value of the differencebetween the pixel value of the pixel in the frame #n−1 and the pixelvalue of the pixel in the frame #n−2 is equal to or smaller than thethreshold value Th, the still/motion judgment unit 202-4 supplies thestill/motion judgment indicating “still” to the region judgment unit203-3.

[0531] In the event that the still/motion judgment supplied from thestill/motion judgment unit 202-1 indicates “still”, and the still/motionjudgment supplied from the still/motion judgment unit 202-2 indicatesmotion, the region judgment unit 203-1 judges that the pixel which isthe object of specifying the region in the frame #n belongs to theuncovered background region, and sets the uncovered background regionjudgment flag corresponding to the judged pixel in the region, to “1”,which indicates that the pixel belongs to the uncovered backgroundregion.

[0532] In the event that the still/motion judgment supplied from thestill/motion judgment unit 202-1 indicates motion, or the still/motionjudgment supplied from the still/motion judgment unit 202-2 indicatesstill, the region judgment unit 203-1 judges that the pixel which is theobject of specifying the region in the frame #n does not belong to theuncovered background region, and sets the uncovered background regionjudgment flag corresponding to the judged pixel in the region to “0”,which indicates that the pixel does not belong to the uncoveredbackground region.

[0533] The region judgment unit 203-1 supplies the uncovered backgroundregion judgment flag which has been set to “1” or “0”, as describedabove, to the judgment flag storing memory 204.

[0534] In the event that the still/motion judgment supplied from thestill/motion judgment unit 202-2 indicates “still”, and the still/motionjudgment supplied from the still/motion judgment unit 202-3 indicates“still”, the region judgment unit 203-2 judges that the pixel which isthe object of specifying the region in the frame #n belongs to the stillregion, and sets the still region judgment flag corresponding to thepixel judged in the region, to “1”, which indicates that the pixelbelongs to the still region.

[0535] In the event that the still/motion judgment supplied from thestill/motion judgment unit 202-2 indicates motion, or the still/motionjudgment supplied from the still/motion judgment unit 202-3 indicatesmotion, the region judgment unit 203-2 judges that the pixel which isthe object of specifying the region in the frame #n does not belong tothe still region, and sets the still region judgment flag correspondingto the judged pixel in the region, to “0”, which indicates that thepixel does not belong to the still region.

[0536] The region judgment unit 203-2 supplies the still region judgmentflag which has been set to “1” or “0” as described above, to judgmentflag storing frame memory 204.

[0537] In the event that the still/motion judgment supplied from thestill/motion judgment unit 202-2 indicates motion, and the still/motionjudgment supplied from the still/motion judgment unit 202-3 indicatesmotion, the region judgment unit 203-2 judges the pixel which is theobject of specifying the region in the frame #n belongs to the movingregion, and sets the moving region judgment flag corresponding to thejudged pixel in the region, to “1”, which indicates that the pixelbelongs to the moving region.

[0538] In the event that the still/motion judgment supplied from thestill/motion judgment unit 202-2 indicates “still”, or the still/motionjudgment supplied from the still/motion judgment unit 202-3 indicates“still”, the region judgment unit 203-2 judges that the pixel which isthe object of specifying the region in the frame #n does not belong tothe moving region, and sets the moving region judgment flagcorresponding to the judged pixel in the region, to “0”, which indicatesthat the pixel does not belong to the moving region.

[0539] The region judgment unit 203-2 supplies the moving regionjudgment flag which has been set to “1” or “0”, to the judgment flagstoring frame memory 204.

[0540] In the event that the still/motion judgment supplied from thestill/motion judgment unit 202-3 indicates motion, and the still/motionjudgment supplied from the still/motion judgment unit 202-4 indicates“still”, the region judgment unit 203-3 judges that the pixel which isthe object of specifying the region in the frame #n belongs to thecovered background region, and sets the covered background regionjudgment flag corresponding to the judged pixel in the region to “1”,which indicates that the pixel belongs to the covered background region.

[0541] In the event that the still/motion judgment supplied from thestill/motion judgment unit 202-3 indicates “still”, or the still/motionjudgment supplied from the still/motion judgment unit 202-4 indicatesmotion, the region judgment unit 203-3 judges that the pixel which isthe object of specifying the region in the frame #n does not belong tothe covered background region, and sets the covered background regionjudgment flag corresponding to the judged pixel in the region to “0”,which indicates that the pixel does not belong to the covered backgroundregion.

[0542] The region judgment unit 203-3 supplies the covered backgroundregion judgment flag which has been set to “1” or “0” as describedabove, to the judgment flag storing frame memory 204.

[0543] The judgment flag storing frame memory 204 stores the uncoveredbackground region judgment flag supplied from the region judgment unit203-1, the still region judgment flag supplied from the region judgmentunit 203-2, the moving region judgment flag supplied from the regionjudgment unit 203-2, and the covered background region judgment flagsupplied from the region judgment unit 203-3.

[0544] The judgment flag storing frame memory 204 supplies the uncoveredbackground region judgment flag, the still region judgment flag, themoving region judgment flag, and the covered background region judgmentflag, which are stored therein, to a synthesizing unit 205. Thesynthesizing unit 205 generates the region information which indicateswhich of the uncovered background region, the still region, the movingregion, or the covered background region, each pixel belongs to, andsupplies the information to judgment flag storing frame memory 206,based upon the uncovered background region judgment flag, the stillregion judgment flag, the moving region judgment flag, and the coveredbackground region judgment flag, which are supplied from the judgmentflag storing frame memory 204.

[0545] The judgment flag storing frame memory 206 stores the regioninformation supplied from the synthesizing unit 205, and also outputsthe stored region information.

[0546] An example for processing performed by the region specifying unit103 will now be described with reference to FIG. 36 through FIG. 40.

[0547] In the event that the object corresponding to the foregroundmoves, the position of the image corresponding to the object on thescreen changes with each frame. As shown in FIG. 36, in the frame #n,the image corresponding to the object which is at the position indicatedby Yn(x,y) is at the position Yn+1(x,y) in the following frame #n+1.

[0548]FIG. 37 is a model diagram wherein the pixel values of pixels ofthe image corresponding to the foreground object, which are adjacentlyarrayed in sequence in a image movement direction, develop over the timedirection. For example, in the event that the image moving directioncorresponding to the foreground object is horizontal to the screen, themodel diagram in FIG. 37 indicates the model wherein the pixel values ofadjacent pixels in one line develop over the time direction.

[0549] In FIG. 37, the line in the frame #n is the same as the line inthe frame #n+1.

[0550] The foreground components corresponding to the object, which areincluded in the second pixel through thirteenth pixel from the left inthe frame #n, are included in the sixth through seventeenth pixels fromthe left in the frame #n+1.

[0551] In the frame #n, the pixels belonging to the covered backgroundregion are the eleventh through thirteenth pixels from the left, and thepixels belonging to the uncovered background region are the secondthrough fourth pixels from the left. In the frame #n+1, the pixelsbelonging to the covered background region are the fifteenth throughseventeenth pixels from the left, and the pixels belonging to theuncovered background region are sixth through eighth pixels from theleft.

[0552] With the example shown in FIG. 37, the movement amount v is 4,since the foreground components included in the frame #n move by fourpixels in the frame #n+1. The virtual dividing number is 4,corresponding to the movement value v.

[0553] Next, a description will be made regarding the change of thepixel values of the pixels belonging to the mixed region in the framesprevious to and following the frame of interest.

[0554] In the frame #n wherein the background keeps still and themovement amount v of the foreground is 4, shown in FIG. 38, the pixelsbelonging to the covered background region are the fifteenth throughseventeenth pixels from the left. Since the movement amount v is 4, inthe previous frame #n 1, the fifteenth through seventeenth pixels fromthe left include only the background components, and belong to thebackground region. Also, in the frame # n−2 which is one further before,the fifteenth through seventeenth pixels from the left contain only thebackground components, and belong to the background region.

[0555] Note that since the object corresponding to the background keepsstill, the pixel value of the fifteenth pixel from the left in the frame#n−1 do not change from the pixel value of the fifteenth pixel from theleft in the frame #n−2. Similarly, the pixel value of the sixteenthpixel from the left in the frame #n−1 do not change from the pixel valueof the sixteenth pixel from the left in the frame #n−2, and the pixelvalues of the seventeenth pixel from the left in the frame #n−1 do notchange from the pixel value of the seventeenth pixel from the left inthe frame #n−2.

[0556] That is to say, the pixels of the frame #n−1 and frame #n−2corresponding to the pixels belonging to the covered background regionin the frame #n consists of only the background components, and thepixel values do not change, and accordingly the absolute value of thedifference therebetween is approximately zero. Accordingly, judgment ismade that the still/motion judgment for the pixels of the frame #n−1 andthe frame #n−2 corresponding to the pixels belonging to the mixed regionin the frame #n is still by the still/motion judgment unit 202-4.

[0557] Since the pixels belonging to the covered background region inthe frame #n contain the foreground components, the pixel values aredifferent from the case wherein the pixel values in the frame #n−1consist of only the background components. Accordingly, judgment is madethat the still/motion judgment for the pixels belonging to the mixedregion in the frame #n and the pixels in the frame #n−1 correspondingthereto is motion by the still/motion judgment unit 202-3.

[0558] As described above, the region judgment unit 203-3 judges thatthe corresponding pixels belong to the covered background region in theevent that the still/motion judgment unit 202-3 supplies the results ofthe still/motion judgment which indicates motion, and the still/motionjudgment unit 202-4 supplies the results of the still/motion judgmentwhich indicates “still”.

[0559] In the frame #n wherein the background keeps still and theforeground movement amount v is 4 as shown in FIG. 39, the pixelsincluded in the uncovered background region are the second throughfourth pixels from the left. Since the movement amount v is 4, in theframe #n+1 following the frame #n, the second through fourth pixels fromthe left include only the background components, and belong to thebackground region. Also, in the frame #n+2 further one frame followingthe frame #n+1, the second through fourth pixels from the left containonly the background components, and belong to the background region.

[0560] Note that since the object corresponding to the background keepsstill, the pixel values of the second pixel from the left in the frame#n+2 does not change from the pixel value of the second pixel from theleft in the frame #n+1. Similarly, the pixel value of the third pixelfrom the left in the frame #n+2 does not change from the pixel value ofthe third pixel from the left in the frame #n+1, and the pixel value ofthe fourth pixel from the left in the frame #n+2 does not change fromthe pixel value of the fourth pixel from the left in the frame #n+1.

[0561] That is to say, the pixels of the frame #n+1 and the frame #n+2,corresponding to the pixels belonging to the uncovered background regionin the frame #n, consist of only the background components, so the pixelvalues thereof do not change, and accordingly the absolute value of thedifference thereof is approximately zero. Accordingly, judgment is madethat the still/motion judgment for the pixels of the frame #n+1 and theframe #n+2 corresponding to the pixels belonging to the mixed region inthe frame #n is “still” by the still/motion judgment unit 202-1.

[0562] Since the pixels belonging to the uncovered background region inthe frame #n contain the foreground components, the pixel values aredifferent from the case wherein the pixels consists of only thebackground components in the frame #n+1. Accordingly, judgment is madethat the still/motion judgment for the pixels belonging to the mixedregion in the frame #n and the pixels corresponding thereto in the frame#n+1 is motion by the still/motion judgment unit 202-2.

[0563] As described above, the region judgment unit 203-1 judges thatthe corresponding pixels belong to the uncovered background region inthe event that the still/motion judgment unit 202-2 supplies the resultsof the still/motion judgment which indicates motion, and thestill/motion judgment unit 202-1 supplies the still/motion judgmentwhich indicates “still”.

[0564]FIG. 40 is a diagram which illustrates judgment conditions of theregion specifying unit 103 in the frame #n. In the event that judgmentis made that the pixel in the frame #n−2 at the same position as theposition of the pixel which is the object of judgment on the image inthe frame #n, and the pixel in the frame #n−1 at the same position asthe position of the pixel which is the object of judgment on the imagein the frame #n, are “still”, and judgment is made that the pixel in theframe #n−1 at the same position as the position of the pixel which isthe object of judgment on the image in the frame #n, and the pixel inthe frame #n are motion, the region specifying unit 103 judges that thepixel which is the object of judgment of the frame #n belongs to thecovered background region.

[0565] In the event that judgment is made that the pixel in the frame#n−1 at the same position as the position of the pixel which is theobject of judgment on the image in the frame #n, and the pixel in theframe #n, are judged to be “still”, and judgment is made that the pixelin the frame #n and the pixel in the frame #n+1 at the same position asthe position of the pixel which is the object of judgment on the imagein the frame #n, are judged to be “still”, the region specifying unit103 judges that the pixel which is the object of judgment of the frame#n belongs to the still region.

[0566] In the event that judgment is made that the pixel in the frame#n−1 at the same position as the position of the pixel which is theobject of judgment on the image in the frame #n, and the pixel in theframe #n, are judged to be motion, and judgment is made that the pixelof the frame #n and the pixel in the frame #n+1 at the same position asthe position of the pixel which is the object of judgment on the imagein the frame #n, are judged to be motion, the region specifying unit 103judges that the pixel which is the object of judgment of the frame #nbelongs to the movement region.

[0567] In the event that judgment is made that the pixel of the frame #nand the pixel in the frame #n+1 at the same position as the position ofthe pixel which is the object of judgment on the image in the frame #n,are motion, and judgment is made that the pixel in the frame #n+1 at thesame position as the position of the pixel which is the object ofjudgment on the image in the frame #n, and the pixel in the frame #n+2at the same position as the position of the pixel which is the object ofjudgment on the image in the frame #n, are judged to be “still”, theregion specifying unit 103 judges that the pixel which is the object ofjudgment of the frame #n belongs to the uncovered background region.

[0568]FIG. 41A through FIG. 41D are diagrams which illustrate examplesof results of the region specifying unit 103 specifying the region. InFIG. 41A, the pixels which have been judged to belong to the coveredbackground region are displayed in white. In FIG. 41B, the pixels whichhave been judged to belong to the uncovered background region aredisplayed in white.

[0569] In FIG. 41C, the pixels which have been judged to belong to themovement region are displayed in white. In FIG. 41D, the pixels whichhave been judged to belong to the still region are displayed in white.

[0570]FIG. 42 is a diagram which illustrates the region information asan image, indicating the mixed region of the region information whichthe judgment flag storing frame memory 206 outputs. In FIG. 42, thepixels which have been judged to belong to the covered background regionor the uncovered background region, i.e., the pixels judged to belong tothe mixed region, are displayed in white. The region informationindicating the mixed region, which the judgment flag storing framememory 206 outputs, indicates the mixed region and the portions whichhave texture within the foreground region and are surrounded by portionswhich have no texture.

[0571] Next, referring to the flowchart in FIG. 43, the processing forregion specifying by the region specifying unit 103 will be described.In Step S201, the frame memory 201 obtains the images of the frame #n−2through the frame #n+2, including the frame #n which is the object ofjudgment.

[0572] In Step S202, the still/motion judgment unit 202-3 judges whetheror not the pixel of the frame #n−1 and the pixel of the frame #n at thesame position keep still, and in the event of judgment of “still”, theflow proceeds to Step S203, and the still/motion judgment unit 202-2judges whether or not the pixel of the frame #n and the pixel of theframe #n+1 at the same position keep still.

[0573] In Step S203, in the event that judgment is made that the pixelof the frame #n and the pixel of the frame #n+1 at the same position are“still”, the flow proceeds to Step S204, and the region judgment unit203-2 sets the still region judgment flag corresponding to the judgedpixel in the region to “1” which indicates the pixel belongs to thestill region. The region judgment unit 203-2 supplies the still regionjudgment flag to the judgment flag storing frame memory 204, and theprocedure proceeds to Step S205.

[0574] In Step S202, in the event that judgment is made that the pixelof the frame #n−1 and the pixel of the frame #n at the same position aremotion, or in Step S203, judgment is made that the pixel of the frame #nand the pixel of the frame #n+1 at the same position are motion, thepixel of the frame #n does not belong to the still region, andaccordingly the processing in Step S204 is skipped, and the procedureproceeds to Step S205.

[0575] In Step S205, the still/motion judgment unit 202-3 judges whetheror not the pixel of the frame #n−1 and the pixel of the frame #n at thesame position are in motion, and in the event of judgment of motion, theflow proceeds to Step S206, and the still/motion judgment unit 202-2judges whether or not the pixel of the frame #n and the pixel of theframe #n+1 at the same position are in motion.

[0576] In Step S206, in the event that judgment is made that the pixelof the frame #n and the pixel of the frame #n+1 at the same position arein motion, the flow proceeds to Step S207, the region judgment unit203-2 set the movement region judgment flag corresponding to the judgedpixel in the region to “1” which indicates that the pixel belongs to themovement region. The region judgment unit 203-2 supplies the movementregion judgment flag to the judgment flag storing frame memory 204, andthe procedure proceeds to Step S208.

[0577] In Step S205, in the event that judgment is made that the pixelof the frame #n−1 and the pixel of the frame #n at the same position are“still”, or in Step S206, in the event that judgment is made that thepixel of the frame #n and the pixel of the frame #n+1 at the sameposition are “still”, since the pixel of the frame #n does not belong tothe movement region, the processing in Step S207 is skipped, and theprocedure proceeds to Step S208.

[0578] In Step S208, the still/motion judgment unit 202-4 judges whetheror not the pixel of the frame #n−2 and the pixel of the frame #n−1 atthe same position keeps still, and in the event of judgment of “still”,the flow proceeds to Step S209, and the still/motion judgment unit 202-3judges whether or not the pixel of the frame #n−1 and the pixel of theframe #n at the same position are in motion.

[0579] In Step S209, in the event that judgment is made that the pixelof the frame #n−1 and the pixel of the frame #n at the same position arein motion, the flow proceeds to Step S210, and the region judgment unit203-3 sets the covered background region judgment flag corresponding tothe judged pixel in the region to “1” which indicates that the pixelbelongs to the covered background region. The region judgment unit 203-3supplies the covered background region judgment flag to the judgmentflag storing frame memory 204, and the procedure proceeds to Step S211.

[0580] In Step S208, in the event that judgment is made that the pixelof the frame #n−2 and the pixel of the frame #n−1 at the same positionare in motion, or in Step S209, in the event that judgment is made thatthe pixel of the frame #n−1 and the pixel of the frame #n at the sameposition are “still”, the pixel of the frame #n does not belong to thecovered background region, so the processing in Step S210 is skipped,and the procedure proceeds to Step S211.

[0581] In Step S211, the still/motion judgment unit 202-2 judges whetheror not the pixel of the frame #n and the pixel of the frame #n+1 at thesame position are in motion, and in the event of judgment of motion, theflow proceeds to Step S212, and the still/motion judgment unit 202-1judges whether or not the pixel of the frame #n+1 and the pixel of theframe #n+2 at the same position keep still.

[0582] In Step S212, in the event that judgment is made that the pixelof the frame #n+1 and the pixel of the frame #n+2 at the same positionare “still”, the flow proceeds to Step S213, and the region judgmentunit 203-1 sets the uncovered background region judgment flagcorresponding to the judged pixel in the region to “1” which indicatesthat the pixel belongs to the uncovered background region. The regionjudgment unit 203-1 supplies the uncovered background region judgmentflag to the judgment flag storing frame memory 204, and the procedureproceeds to Step S214.

[0583] In Step S211, in the event that judgment is made that the pixelof the frame #n and the pixel of the frame #n+1 at the same position are“still”, or in Step 212, in the event that judgment is made that thepixel of the frame #n+1 and the pixel of the frame #n+2 at the sameposition are in motion, since the pixel of the frame #n does not belongto the uncovered background region, the processing in Step S213 isskipped, and the procedure proceeds to Step S214.

[0584] In Step 214, the region specifying unit 103 judges whether or notall the pixels in the frame #n are region-specified, and in the eventthat judgment is made that not all pixels are region-specified, theprocedure returns to Step S202, and repeats the processing of specifyingthe region for other pixels.

[0585] In Step S214, in the event that judgment is made that all thepixels in the frame #n are region-specified, the flow proceeds to StepS215, and the synthesizing unit 205 generates the region informationwhich indicates the mixed region based upon the uncovered backgroundregion judgment flag and the covered background region judgment flag,which are stored in the judgment flag storing frame memory 204, andfurthermore generates the region information which indicates which ofthe uncovered background region, the still region, the movement region,or the covered background region, each pixel belongs to, sets thegenerated region information for the judgment flag storing frame memory206, and the processing ends.

[0586] As described above, the region specifying unit 103 can generateregion information which indicates which of the movement region, thestill region, the uncovered background region, or the covered backgroundregion, each pixel contained in the frame belongs to.

[0587] Note that an arrangement may be made wherein the regionspecifying unit 103 generates the region information corresponding tothe mixed region and the region information made up of flags whichindicates which of the movement region, the still region, or the mixedregion, each of pixels contained in the frame belongs to, by applyingthe logical sum to the region information corresponding to the uncoveredbackground region and the covered background region.

[0588] In the event that the object corresponding to the foreground hastexture, the region specifying unit 103 can specify the movement regionmore accurately.

[0589] The region specifying unit 103 can output the region informationindicating the movement region as the region information indicating theforeground region, or output the region information indicating the stillregion as the region information indicating the background region.

[0590] While description has been made wherein the object correspondingto the background keeps still, the processing of specifying the regiondescribed above can be applied even if the image corresponding to thebackground region contains motion. For example, in the event that theimage corresponding to the background region moves in a constant manner,the region specifying unit 103 shifts the entire image corresponding tothe movement, and performs processing in the same manner as with thecase wherein the object corresponding to the background keeps still.Also, in the event that the image corresponding to the background regioncontains a different motion at each local position, the regionspecifying unit 103 selects the pixel corresponding to the motion, andperforms the above-described processing.

[0591]FIG. 44 is a block diagram which illustrates another example ofthe structure of the region specifying unit 103. The region specifyingunit 103 shown in FIG. 44 does not use movement vectors. A backgroundimage generating unit 301 generates the background image correspondingto the input image, and supplies the generated background image to abinary object image extracting unit 302. The background image generatingunit 301 extracts, for example, the image object corresponding to thebackground object contained in the input image, and generates thebackground image.

[0592] An example of a model diagram is illustrated in FIG. 45 whereinthe pixel values of the pixels arrayed in sequence adjacently in amovement direction of the image corresponding to the foreground object,develop over the time direction. For example, in the event that themovement direction of the image corresponding to the foreground objectis horizontal to the screen, the model diagram in FIG. 45 illustrates amodel wherein the pixel values of the adjacent pixels in one linedevelop over the time direction.

[0593] In FIG. 45, the line in the frame #n is the same as the line inthe frame #n−1 and the line in the frame #n+1.

[0594] In the frame #n, the foreground components corresponding to theobject, which are contained in the sixth pixel through seventeenth pixelfrom the left, are contained in the second through thirteenth pixelsfrom the left in the frame #n−1, and are contained in the tenth throughtwenty-first pixels from the left in the frame #n+1.

[0595] In the frame #n−1, the pixels belonging to the covered backgroundregion are the eleventh through thirteenth pixels from the left, and thepixels belonging to the uncovered background region are the secondthrough fourth pixels from the left. In the frame #n, the pixelsbelonging to the covered background region are the fifteenth through theseventeenth pixels from the left, and the pixels belonging to theuncovered background region are the sixth through eighth pixels from theleft. In the frame #n+1, the pixels belonging to the covered backgroundregion are the nineteenth through twenty-first pixels from the left, andthe pixels belonging to the uncovered background region are the tenththrough twelfth pixels from the left.

[0596] In the frame #n−1, the pixels belonging to the background regionare the first from the left, and the fourteenth through twenty-firstpixels from the left. In the frame #n, the pixels belonging to thebackground region are the first through fifth pixels from the left, andthe eighteenth through twenty-first pixels from the left. In the frame#n+1, the pixels belonging to the background region are the firstthrough ninth pixels from the left.

[0597] An example of the background image corresponding to the exampleshown in FIG. 45, which is generated by the background image generatingunit 301, is illustrated in FIG. 46. The background image is made up ofthe pixels corresponding to the background object, and does not containimage components corresponding to the foreground object.

[0598] The binary object image extracting unit 302 generates a binaryobject image based upon the correlation between the background image andthe input image, and supplies the generated binary object image to atime change detecting unit 303.

[0599]FIG. 47 is a block diagram which illustrates the configuration ofthe binary object image extracting unit 302. A correlation valuecomputing unit 321 computes the correlation between the background imagesupplied from the background image generating unit 301 and the inputimage, generates a correlation value, and supplies the generatedcorrelation value to a threshold value processing unit 322.

[0600] The correlation value computing unit 321 applies Expression (4)to a block 3×3 wherein X₄ is centered in the background image as shownin FIG. 48A, and a block 3×3 wherein Y₄ corresponding to the block inthe background image is centered in the input image as shown in FIG.48B, and calculates a correlation value corresponding to the Y₄, forexample. $\begin{matrix}{\begin{matrix}{Correlation} \\{Value}\end{matrix} = \frac{\sum\limits_{i = 0}^{8}\quad {\left( {X_{i} - \overset{\_}{X}} \right){\sum\limits_{i = 0}^{8}\quad \left( {Y_{i} - \overset{\_}{Y}} \right)}}}{\sqrt{\sum\limits_{i = 0}^{8}\quad {\left( {X_{i} - \overset{\_}{X}} \right)^{2} \cdot {\sum\limits_{i = 0}^{8}\quad \left( {Y_{i} - \overset{\_}{Y}} \right)^{2}}}}}} & (4) \\{\overset{\_}{X} = \frac{\sum\limits_{i = 0}^{8}\quad X_{i}}{9}} & (5) \\{\overset{\_}{Y} = \frac{\sum\limits_{i = 0}^{8}\quad Y_{i}}{9}} & (6)\end{matrix}$

[0601] The correlation value computing unit 321 supplies the correlationvalue calculated corresponding to each pixel as described above to thethreshold value processing unit 322.

[0602] Also, an arrangement may be made wherein the correlation valuecomputing unit 321, for example, applies Expression (7) to the block 3×3in the background image wherein X₄ is centered as shown in FIG. 49A, andthe block 3×3 in the input image wherein Y₄ is centered corresponding tothe block in the background image, and calculates the sum of absolutevalue of difference corresponding to Y₄. $\begin{matrix}{{\begin{matrix}{{Sum}\quad {of}\quad {Absolute}} \\{{Value}\quad {of}\quad {Difference}}\end{matrix} = {\sum\limits_{i = 0}^{8}{\left( {X_{i} - Y_{i}} \right)}}}\quad} & (7)\end{matrix}$

[0603] The correlation value computing unit 321 supplies the differenceabsolute value calculated as described above as the correlation value tothe threshold value processing unit 322.

[0604] The threshold value processing unit 322 compares the pixel valueof the correlation image with the threshold value th0, and in the eventthat the correlation value is equal to or less than the threshold valueth0, the threshold value processing unit 322 sets the pixel value of thebinary object image to 1, and in the event that the correlation value isgreater than the threshold value th0, the threshold value processingunit 322 sets the pixel value of the binary object image to 0, andoutputs the binary object image of which each pixel value has been setto 0 or 1. The threshold value processing unit 322 may store thethreshold value th0 beforehand, and may use the threshold value th0which is input externally.

[0605]FIG. 50 is a diagram which illustrates an example of the binaryobject image corresponding to the model of the input image shown in FIG.45. In the binary object image, a pixel value of a pixel having a highcorrelation with the background image is set to 0.

[0606]FIG. 51 is a block diagram which illustrates the configuration ofthe time change detecting unit 303. Frame memory 341 stores the binaryobject images of the frame #n−1, frame #n, and frame #n+1, supplied fromthe binary object image extracting unit 302 at the point of judgment ofthe region for the pixel of the frame #n.

[0607] A region judgment unit 342 judges the region for each pixel ofthe frame #n based upon the binary object images of the frame #n−1,frame #n, and frame #n+1, which are stored in the frame memory 341,generates the region information, and outputs the generated regioninformation.

[0608]FIG. 52 is a diagram which describes the judgment made by theregion judgment unit 342. In the event that the pixel of interest of thebinary object image of the frame #n is 0, the region judgment unit 342judges the pixel of interest of the frame #n to belong to the backgroundregion.

[0609] In the event that the pixel of interest of the binary objectimage of the frame #n is 1, the corresponding pixel of the binary objectimage of the frame #n−1 is 1, and the corresponding pixel of the binaryobject image of the frame #n+1 is 1, the region judgment unit 342 judgesthe pixel of interest of the frame #n to belong to the foregroundregion.

[0610] In the event that the pixel of interest of the binary objectimage of the frame #n is 1, and the corresponding pixel of the binaryobject image of the frame #n−1 is 0, the region judgment unit 342 judgesthe pixel of interest of the frame #n to belong to the coveredbackground region.

[0611] In the event that the pixel of interest of the binary objectimage of the frame #n is 1, and the corresponding pixel of the binaryobject image of the frame #n+1 is 0, the region judgment unit 342 judgesthe pixel of interest of the frame #n to belong to the uncoveredbackground region.

[0612]FIG. 53 is a diagram which illustrates an example wherein the timechange detecting unit 303 judges the binary object image correspondingto the model of the input image shown in FIG. 45. The time changedetecting unit 303 judges the first through fifth pixels from the leftof the frame #n to belong to the background region since thecorresponding pixels of the binary object image of the frame #n are 0.

[0613] The time change detecting unit 303 judges the sixth through ninthpixels from the left to belong to the uncovered background region sincethe pixels of the binary object image of the frame #n are 1, and thecorresponding pixels of the frame #n+1 are 0.

[0614] The time change detecting unit 303 judges the tenth throughthirteenth pixels from the left to belong to the foreground region sincethe pixels of the binary object image of the frame #n are 1, thecorresponding pixels of the frame #n−1 are 1, and the correspondingpixels of the frame #n+1 are 1.

[0615] The time change detecting unit 303 judges the fourteenth throughseventeenth pixels from the left to belong to the covered backgroundregion since the pixels of the binary object image of the frame #n are1, and the corresponding pixels of the frame #n−1 are 0.

[0616] The time change detecting unit 303 judges the eighteenth throughtwenty-first pixels from the left to belong to the background regionsince the corresponding pixels of the binary object image of the frame#n are 0.

[0617] The processing of specifying the region by the region judgmentunit 103 will be now described, referring to the flowchart shown in FIG.54. In Step S301, the background image generating unit 301 of the regionjudgment unit 103, for example, generates the background image byextracting the image object corresponding to the background objectcontained in the input image based upon the input image, and suppliesthe generated background image to the binary object image extractingunit 302.

[0618] In Step S302, the binary object image extracting unit 302computes the correlation value between the input image and thebackground image supplied from the background image generating unit 301by the computation described referring to FIG. 48A and FIG. 48B, forexample. In Step S303, the binary object image extracting unit 302computes the binary object image from the correlation value and thethreshold value th0 by comparing the correlation value with thethreshold value th0, for example.

[0619] In Step S304, the time change detecting unit 303 performsprocessing of region judgment, and the processing ends.

[0620] The processing of the region judgment corresponding to Step S304will be described in detail, referring to the flowchart shown in FIG.55. In Step S321, the region judgment unit 342 of the time changedetecting unit 303 judges whether or not the pixel of interest in theframe #n stored in the frame memory 341 is 0, and in the event that thejudgment is made that the pixel of the interest in the frame #n is 0,the flow proceeds to Step S322, makes settings to the effect that thepixel of interest in the frame #n belongs to the background region, andthe processing ends.

[0621] In Step S321, in the event that judgment is made that the pixelof interest in the frame #n is 1, the flow proceeds to Step S323, andthe region judgment unit 342 of the time change detecting unit 303judges whether or not the pixel of interest in the frame #n stored inthe frame memory 341 is 1, and the corresponding pixel in the frame #n−1is 0, and in the event that judgment is made that the pixel of interestin the frame #n is 1, and the corresponding pixel in the frame #n−1 is0, the flow proceeds to Step S324, makes settings to the effect that thepixel of interest in the frame #n belongs to the covered backgroundregion, and the processing ends.

[0622] In Step S323, in the event that judgment is made that the pixelof interest in the frame #n is 0, or the corresponding pixel in theframe #n−1 is 1, the flow proceeds to Step S325, and the region judgmentunit 342 of the time change detecting unit 303 judges whether or not thepixel of interest in the frame #n stored in the frame memory 341 is 1,and the corresponding pixel in the frame #n+1 is 0, and in the eventthat judgment is made that the pixel of interest in the frame #n is 1,and the corresponding pixel in the frame #n+1 is 0, the flow proceeds toStep S326, makes settings to the effect that the pixel of interest inthe frame #n belongs to the uncovered background region, and theprocessing ends.

[0623] In Step 325, in the event that judgment is made that the pixel ofinterest in the frame #n is 0, or the corresponding pixel in the frame#n+1 is 1, the flow proceeds to Step S327, and the region judgment unit342 of the time change detecting unit 303 sets the pixel of interest inthe frame #n for the foreground region, and the processing ends.

[0624] As described above, the region specifying unit 103 can specifywhich of the foreground region, the background region, the coveredbackground region, or the uncovered background region, the pixel of theinput image belongs to, and can generate region informationcorresponding to the specified results.

[0625]FIG. 56 is a block diagram which illustrates another configurationof the region specifying unit 103. The region specifying unit 103 shownin FIG. 56 uses the movement vector and the position informationthereof, which are supplied from the movement detecting unit 102.Portions the same as those shown in FIG. 44 are denoted by the samereference numerals, and description thereof will be omitted.

[0626] A robustification unit 361 generates a robustified binary objectimage based upon N frames of the binary object image supplied from thebinary object image extracting unit 302, and outputs to the time changedetecting unit 303.

[0627]FIG. 57 is a block diagram which describes the configuration ofthe robustification unit 361. A movement compensation unit 381compensates for the movement of the binary object image of N framesbased upon the movement vector and the position information thereofsupplied from the movement detecting unit 102, and outputs the binaryobject image which has been subjected to compensation of movement to aswitch 382.

[0628] The movement compensation of the movement compensation unit 381will be described with reference to examples shown in FIG. 58 and FIG.59. For example, in cases wherein the region in the frame #n is judged,in the event that there is input of the binary object images of theframe #n−1, the frame #n, and the frame #n+1, shown by way of theexample in FIG. 58, the movement compensation unit 381 compensates formovement of the binary object image of the frame #n−1 and the binaryobject image of the frame #n+1, based upon the movement vector suppliedfrom the movement detecting unit 102, and supplies the binary objectimage which has been subjected to compensation of movement to the switch382, as indicated in the example shown in FIG. 59.

[0629] The switch 382 outputs the binary object image which has beensubjected to movement compensation of the first frame, to the framememory 383-1, and outputs the binary object image which has beensubjected to movement compensation of the second frame to the framememory 383-2. Similarly, the switch 382 outputs each of the binaryobject images of which the third through N−1'th frames have beensubjected to compensation for the movement to each of frame memory 383-3through frame memory 383-(N−1), respectively, and outputs the binaryobject image of which the N'th frame has been subjected to movementcompensation to frame memory 383-N.

[0630] The frame memory 383-1 stores the binary object image of whichthe first frame has been subjected to movement compensation, and outputsthe stored binary object image to a weighting addition unit 384-1. Theframe memory 383-2 stores the binary object image of which the secondframe has been subjected to movement compensation, and outputs thestored binary object image to a weighting addition unit 384-2.

[0631] Similarly, each of the frame memory 383-3 through the framememory 383-(N−1) stores each of the binary object images of which one ofthe third frame through N−1'th frame has been subjected to compensationfor the movement, and outputs the stored binary object image to each ofthe weighing addition unit 384-3 through the weighing addition unit384-(N−1). The frame memory 383-N stores the binary object image ofwhich N'th frame has been subjected to compensation for the movement,and outputs the stored binary object image to a weighing addition unit384-N.

[0632] The weighing addition unit 384-1 multiplies the pixel value ofthe binary object image of which the first frame has been subjected tocompensation for the movement supplied from the frame memory 383-1 bythe predetermined weight w1, and supplies to an accumulation unit 385.The weighing addition unit 384-2 multiplies the pixel value of thebinary object image of the second frame which has been subjected tomovement compensation supplied from the frame memory 383-2 by thepredetermined weight w2, and supplies to an accumulation unit 385.

[0633] Similarly, each of the weighting addition unit 384-3 through theweighing addition unit 384-(N−1) multiplies the pixel value of thebinary object image of one of the third through N−1'th frames, which hasbeen subjected to movement compensation supplied from one of the framememory 383-3 through the frame memory 383-(N−1) by one of thepredetermined weights w3 through w(N−1), and supplies to theaccumulation unit 385. A weighing addition unit 384-N multiplies thepixel value of the binary object image of the N'th frame supplied fromthe frame memory 383-N which has been subjected to movement compensationby the predetermined weight wN, and supplies to the accumulation unit385.

[0634] The accumulation unit 385 accumulates the pixel valuecorresponding to the binary object image, wherein each of the firstthrough N'th frames which has been subjected to movement compensation ismultiplied by one of the predetermined weights w1 through wN, andgenerates the binary object image by comparing the accumulated pixelvalue with the predetermined threshold value th0.

[0635] As described above, the robustification unit 361 generates therobustified binary object image from the N frames of binary objectimages, and supplies to the time change detecting unit 303, so theregion specifying unit 103 of which the configuration is shown in FIG.56 can specify the region more accurately as compared with the caseshown in FIG. 44, even if the input image contains noise.

[0636] The processing for specifying the region of the region specifyingunit 103 of which the configuration is shown in FIG. 56 will now bedescribed, referring to the flowchart shown in FIG. 60. The processingin Step S341 through Step S343 is the same as Step S301 through StepS303 described in the flowchart shown in FIG. 54, respectively, andaccordingly, description thereof will be omitted.

[0637] In Step S344, the robustification unit 361 performs processingfor robustification.

[0638] In Step S345, the time change detecting unit 303 performsprocessing for specifying the region, and the processing ends. Detailsof the processing in Step S345 are the same as the processing describedwith reference to the flowchart shown in FIG. 55, so description thereofwill be omitted.

[0639] Referring to the flowchart shown in FIG. 61, processing ofrobustification corresponding to the processing in Step S344 shown inFIG. 60 will now be described in detail. In Step S361, the movementcompensation unit 381 performs movement compensation processing of theinput binary object image based upon the movement vector and theposition information thereof supplied from the movement detecting unit102. In Step S362, one of the frame memory 383-1 through the framememory 383-N stores the binary object image, which has been subjected tomovement compensation, supplied via the switch 382.

[0640] In Step S363, the robustification unit 361 judges whether or notN binary object images are stored, and in the event that judgment ismade that N binary object images have not been stored, the flow returnsto Step S361, and the robustification unit 363 repeats processing ofcompensation for movement of the binary object image, and processing ofstoring the binary object image.

[0641] In Step S363, in the event that judgment is made that N binaryobject images stored, the flow proceeds to Step S364, and each of theweighting addition units 384-1 through 384-N multiplies each of N binaryobject images, by one of the weights w1 through wN for weighting.

[0642] In Step S365, the accumulation unit 385 accumulates the Nweighted binary object images.

[0643] In Step 366, the accumulation unit 385 generates the binaryobject image from the accumulated image, by comparing with thepredetermined threshold value th1, for example, and the processing ends.

[0644] As described above, the region specifying unit 103 of which theconfiguration is shown in FIG. 56 can generate the region informationbased upon the robustified binary object image.

[0645] As described above, the region specifying unit 103 can generatethe region information which indicates which of the movement region, thestill region, the uncovered background region, or the covered backgroundregion, each of the pixels contained in the frame belongs to.

[0646]FIG. 62 is a block diagram which illustrates an example of theconfiguration of the mixture ratio calculation unit 104. An estimatedmixture ratio processing unit 401 calculates estimated mixture ratio foreach pixel by computation corresponding to a model of a coveredbackground region based upon the input image, and supplies thecalculated estimated mixture ratio to a mixture ratio determination unit403.

[0647] An estimated mixture ratio processing unit 402 calculatesestimated mixture ratio for each pixel by computation corresponding to amodel of the uncovered background region based upon the input image, andsupplies the calculated estimated mixture ratio to the mixture ratiodetermination unit 403.

[0648] Since an assumption may be made that the object corresponding tothe foreground moves at a constant velocity within a shutter period, themixture ratio α of a pixel belonging to the mixed region has a naturesuch as described below. That is to say, the mixture ratio α changeslinearly corresponding to the change of the position of the pixel.

[0649] Taking the change of the pixel position to be one-dimensional,the change of the mixture ratio α may be represented by a straight line,and taking the change of the pixel position to be two-dimensional, thechange of the mixture ratio α may be represented by a plane.

[0650] Note that the period of one frame is short, an assumption may bemade that the object corresponding to the foreground is a rigid body,and moves at a constant velocity.

[0651] In this case, the inclination of the mixture ratio α is inverselyproportionate to the movement amount v of the foreground within theshutter period.

[0652] An example of an ideal mixture ratio α is shown in FIG. 63. Theinclination l of an ideal mixture ratio α in the mixed region may berepresented by the reciprocal of the movement amount v.

[0653] As shown in FIG. 63, an ideal mixture ratio α has a value of 1 inthe background region, and has a value of 0 in the foreground region,and has a value which exceeds 0 and is less than 1 in the mixed region.

[0654] With the example shown in FIG. 64, the pixel value C06 of theseventh pixel from the left in the frame #n may be represented inExpression (8), using the pixel value P06 of the seventh pixel from theleft in the frame #n−1. $\begin{matrix}\begin{matrix}{{C06} = {{{B06}/v} + {{B06}/v} + {{F01}/v} + {{F02}/v}}} \\{= {{{P06}/v} + {{P06}/v} + {{F01}/v} + {{F02}/v}}} \\{= {{{2/v} \cdot {P06}} + {\underset{i = 1}{\sum\limits^{2}}F_{i/v}}}}\end{matrix} & (8)\end{matrix}$

[0655] In Expression (8), the pixel value C06 is represented as thepixel value M of the pixel in the mixed region, and the pixel value P06is represented as the pixel value B of the pixel in the backgroundregion. That is to say, the pixel value M of the pixel in the mixedregion and the pixel value B of the pixel in the background region maybe represented as in Expression (9) and Expression (10), respectively.

M=C06  (9)

B=P06  (10)

[0656] In Expression (8), 2/v corresponds to the mixture ratio α. Sincethe movement amount v is 4, the mixture ratio α of the seventh pixelfrom the left in the frame #n is 0.5.

[0657] As described above, Expression (3) indicating the mixture ratio αmay be rewritten as with Expression (11) by reckoning the pixel value Cin the frame #n of interest to be a pixel value in the mixed region, andreckoning the pixel value P in the frame #n−1 previous to the frame #nto be a pixel value of the background region.

C=α·P+f  (11)

[0658] In Expression (11), f is the sum of the foreground componentscontained in the pixel of interest, Σ_(i)Fi/v.

[0659] The variables included in Expression (11) are two, i.e., themixture ratio α and the sum of the foreground components f.

[0660] In the same way, FIG. 65 illustrates a model wherein the pixelvalues wherein the movement amount v is 4, and virtual dividing numberis 4 in the uncovered background region, develop over the timedirection.

[0661] Expression (3) indicating the mixture ratio α may be representedas in Expression (12) with the pixel value C in the frame #n of interestas a pixel value in the mixed region, and with the pixel value N in theframe #n+1 following the frame #n as a pixel value in the backgroundregion, in the same manner as the representation in the coveredbackground region described above, in the uncovered background region.

C=α·N+f  (12)

[0662] Note that while description has been made with an assumption thatthe background object keeps still, Expression (8) through Expression(12) may be applied by using the pixel values of the pixels at thepositions corresponding to the background movement amount v, even if thebackground object moves. For example, in FIG. 64, in the event that themovement amount v of the object corresponding to the background is 2,the virtual dividing number is 2, and the object corresponding to thebackground moves to the right side in the drawing, the pixel value B ofthe pixel in the background region in Expression (10) is the pixel valueP04.

[0663] Expression (11) and Expression (12) include two variables,respectively, and accordingly the mixture ratio α can not be obtained inthis state. Here, images generally have great correlation spatially, theadjacent pixels have approximately the same value.

[0664] Thus, since the foreground components have great correlationspatially, the mixture ratio is obtained by transforming the expressionso as to obtain the sum of the foreground components from the previousor following frame.

[0665] The pixel value Mc of the seventh pixel from the left in theframe #n in FIG. 66 may be represented in Expression (13).$\begin{matrix}{M_{c} = {{\frac{2}{v} \cdot {B06}} + {\underset{i = 11}{\sum\limits^{12}}F_{i/v}}}} & (13)\end{matrix}$

[0666] The first argument 2/v of the right side in Expression (13)corresponds to the mixture ratio α. The second argument of the rightside in Expression (13) is represented as in Expression (14) using thepixel values in the following frame #n+1. $\begin{matrix}{{\underset{i = 11}{\sum\limits^{12}}F_{i/v}} = {\beta \cdot {\underset{i = 7}{\sum\limits^{10}}F_{i/v}}}} & (14)\end{matrix}$

[0667] Here, an assumption may be made that Expression (15) holds, usingthe spatial correlation of the foreground components.

F=F05=F06=F07=F08=F09=F10=F11=F12  (15)

[0668] Expression (14) may be rewritten as Expression (16) usingExpression (15) $\begin{matrix}\begin{matrix}{{\underset{i = 11}{\sum\limits^{12}}F_{i/v}} = {\frac{2}{v} \cdot F}} \\{= {\beta \cdot \frac{4}{v} \cdot F}}\end{matrix} & (16)\end{matrix}$

[0669] As a result, β may be represented in Expression (17).

β=2/4  (17)

[0670] In general, in the event that an assumption is made wherein theforeground components correlated to the mixed region are the same asshown in Expression (15), Expression (18) is formed by the relationshipof the internal dividing ratio for all the pixel in the mixed region.

β=1−α  (18)

[0671] In the event that Expression (18) holds, Expression (11) maydevelop as indicated in Expression (19). $\begin{matrix}\begin{matrix}{C = {{\alpha \cdot P} + f}} \\{= {{\alpha \cdot P} + {\left( {1 - \alpha} \right) \cdot {\sum\limits_{i = \gamma}^{\gamma + V - I}\quad F_{i/v}}}}} \\{= {{\alpha \cdot P} + {\left( {1 - \alpha} \right) \cdot N}}}\end{matrix} & (19)\end{matrix}$

[0672] Similarly, in the event that Expression (18) holds, Expression(12) may develop as indicated in Expression (20). $\begin{matrix}\begin{matrix}{C = {{\alpha \cdot N} + f}} \\{= {{\alpha \cdot N} + {\left( {1 - \alpha} \right) \cdot {\sum\limits_{i = \gamma}^{\gamma + V - I}\quad F_{i/v}}}}} \\{= {{\alpha \cdot N} + {\left( {1 - \alpha} \right) \cdot P}}}\end{matrix} & (20)\end{matrix}$

[0673] In Expression (19) and Expression (20), since C, N, and P areknown pixel values, the variable included in Expression (19) andExpression (20) is only the mixture ratio α. The relationship between C,N, and P in Expression (19) and Expression (20) is illustrated in FIG.67. C is the pixel value of the pixel of interest in the frame #n forcalculating the mixture ratio α. N is the pixel value of the pixel inthe frame #n+1, of which the position in the spatial directioncorresponds to that of the pixel of interest. P is the pixel value ofthe pixel in the frame #n−1, of which the position in the spatialdirection corresponds to that of the pixel of interest.

[0674] Accordingly, since Expression (19) and Expression (20) includeone variable each, the mixture ratio α can be calculated using the pixelvalues in three frames. The conditions for calculating an accuratemixture ratio α by solving Expression (19) and Expression (20) are that;the foreground components with regard to the mixed region are the same,that is to say, in the foreground image object which has been taken inthe state of the foreground object being still, the pixel values ofpixels of a number twice the movement amount v, which are arrayedsequentially at the boundary of the image object, corresponding to themovement direction of the foreground object, are constant.

[0675] As described above, the mixture ratio α of the pixel belonging tothe covered background region is calculated by Expression (21), and themixture ratio α belonging to the uncovered background region iscalculated by Expression (22).

α=(C−N)/(P−N)  (21)

α=(C−P)/(N−P)  (22)

[0676]FIG. 68 is a block diagram which illustrates the configuration ofthe estimated mixture ratio processing unit 401. Frame memory 421 storesthe input image in increments of frames, and supplies the framefollowing the frame which is input as an input image, to frame memory422 and a mixture ratio computation unit 423.

[0677] The frame memory 422 stores the input image in increments offrames, and supplies the frame following the frame supplied from theframe memory 421, to the mixture ratio computation unit 423.

[0678] Accordingly, in the event that the frame #n+1 is input as aninput image to the mixture ratio computation unit 423, the frame memory421 supplies the frame #n to the mixture ratio computation unit 423, andthe frame memory 422 supplies the frame #n−1 to the mixture ratiocomputation unit 423.

[0679] The mixture ratio computation unit 423 calculates the estimatedmixture ratio of the pixel of interest by the computation represented inExpression (21) based upon the pixel value C of the pixel of interest inthe frame #n, the pixel value N of the pixel in the frame #n+1 whereinthe spatial position thereof corresponds to that of the pixel ofinterest, and the pixel value P of the pixel in the frame #n−1 whereinthe spatial position thereof corresponds to that of the pixel ofinterest, and outputs the calculated estimated mixture ratio. Forexample, in the event that the background keeps still, the mixture ratiocomputation unit 423 calculates the estimated mixture ratio of the pixelof interest based upon the pixel value C of the pixel of interest in theframe #n, the pixel value N of the pixel in the frame #n+1 at the sameposition in the frame as the pixel of interest, and the pixel value P ofthe pixel in the frame #n−1 at the same position in the frame as thepixel of interest, and outputs the calculated estimated mixture ratio.

[0680] As described above, the estimated mixture ratio processing unit401 can calculate the estimated mixture ratio based upon the inputimage, and supply to the mixture ratio decision unit 403.

[0681] Note that the processing of the estimated mixture ratioprocessing unit 402 is the same as that of the estimated mixture ratioprocessing unit 401 except for the processing wherein, while theestimated mixture ratio processing unit 401 calculates the estimatedmixture ratio of the pixel of interest by the computation represented inExpression (21), the estimated mixture ratio processing unit 402calculates the estimated mixture ratio of the pixel of interest by thecomputation represented in Expression (22), and accordingly, descriptionthereof will be omitted.

[0682]FIG. 69 is a diagram which illustrates an example of the estimatedmixture ratio calculated by the estimated mixture ratio processing unit401. The estimated mixture ratio shown in FIG. 69 indicates the resultsin a case wherein the foreground movement amount v corresponding to theobject which moves at a constant velocity is 11, for one line.

[0683] It can be understood that the estimated mixture ratio changesapproximately linearly in the mixed region, as shown in FIG. 63.

[0684] Returning to FIG. 62, the mixture ratio decision unit 403 setsthe mixture ratio α based upon the region information indicating whichof the foreground region, the background region, the covered backgroundregion, or the uncovered background region, the pixel which is theobject of calculation of the mixture ratio α belongs to, supplied fromthe region specifying unit 103. In the event that the pixel which is theobject belongs to the foreground region, the mixture ratio decision unit403 sets the mixture ratio α to 0, in the event that the pixel which isthe object belongs to the background region, sets the mixture ratio α to1, in the event that the pixel which is the object belongs to thecovered background region, sets the mixture ratio α to the estimatedmixture ratio supplied from the estimated mixture ratio processing unit401, and in the event that the pixel which is the object belongs to theuncovered background region, sets the mixture ratio α to the estimatedmixture ratio supplied from the estimated mixture ratio processing unit402. The mixture ratio decision unit 403 outputs the mixture ratio αwhich has been set based upon the region information.

[0685]FIG. 70 is a block diagram which illustrates another configurationof the mixture ratio calculation unit 104. A selection unit 441 suppliesthe pixels belonging to the covered background region and thecorresponding pixels in the following and previous frames, to anestimated mixture ratio processing unit 442, based upon the regioninformation supplied from the region specifying unit 103. The selectionunit 441 supplies the pixels belonging to the uncovered backgroundregion and the corresponding pixels in the previous and followingframes, to an estimated mixture ratio processing unit 443, based uponthe region information supplied from the region specifying unit 103.

[0686] The estimated mixture ratio processing unit 442 calculates theestimated mixture ratio of the pixel of interest belonging to thecovered background region by the computation represented in Expression(21) based upon the pixel values input from the selection unit 441, andsupplies the calculated estimated mixture ratio to a selection unit 444.

[0687] The estimated mixture ratio processing unit 443 calculates theestimated mixture ratio of the pixel of interest belonging to theuncovered background region by the computation represented in Expression(22) based upon the pixel values input from the selection unit 441, andsupplies the calculated estimated mixture ratio to the selection unit444.

[0688] In the event that the pixel which is the object belongs to theforeground region, the selection unit 444 selects the estimated mixtureratio of 0, and sets for the mixture ratio α, and in the event that thepixel which is the object belongs to the background region, theselection unit 444 selects the estimated mixture ratio of 1, and setsfor the mixture ratio α, based upon the region information supplied fromthe region specifying unit 103. In the event that the pixel which is theobject belongs to the covered background region, the selection unit 444selects the estimated mixture ratio supplied from the estimated mixtureratio processing unit 442, and sets for the mixture ratio α, and in theevent that the pixel which is the object belongs to the uncoveredbackground region, the selection unit 444 selects the estimated mixtureratio supplied from the estimated mixture ratio processing unit 443, andsets this for the mixture ratio α. The selection unit 444 outputs themixture ratio α which has been selected and set based upon the regioninformation.

[0689] As described above, the mixture ratio calculation unit 104 havinganother configuration shown in FIG. 70 can calculate the mixture ratio αfor each pixel included in the image, and output the calculated mixtureratio α.

[0690] Referring to the flowchart shown in FIG. 71, the processing forcalculation of the mixture ratio α by the mixture ratio calculation unit104 of which configuration is shown in FIG. 62 will be described. InStep S401, the mixture ratio calculation unit 104 obtains the regioninformation supplied from the region specifying unit 103. In Step S402,the estimated mixture ratio processing unit 401 performs processing ofcomputation of the estimated mixture ratio by a model corresponding tothe covered background region, and supplies the calculated estimatedmixture ratio to the mixture ratio decision unit 403. Details of theprocessing for computation of the estimated mixture ratio will bedescribed later with reference to the flowchart shown in FIG. 72.

[0691] In Step S403, the estimated mixture ratio processing unit 402performs the processing of the computation of the estimated mixtureratio by a model corresponding to the uncovered background region, andsupplies the calculated mixture ratio to the mixture ratio decision unit403.

[0692] In Step S404, the mixture ratio calculation unit 104 judgeswhether or not the mixture ratio α has been estimated for the entireframe, and in the event that judgment is made that the mixture ratio αhas not been estimated for the entire frame, the flow returns to StepS402, and performs the processing of estimation of the mixture ratio αfor the following pixel.

[0693] In the event that judgment is made in Step S404 that the mixtureratio α has been estimated for the entire frame, the flow proceeds toStep S405, and the mixture ratio decision unit 403 sets the mixtureratio α based upon the region information which indicates which of theforeground region, the background region, the covered background region,or the uncovered background region, the pixel belongs to, supplied fromthe region specifying unit 103. In the event that the pixel which is theobject belongs to the foreground region, the mixture ratio decision unit403 sets the mixture ratio α to 0, in the event that the pixel which isthe object belongs to the background region, sets the mixture ratio α to1, in the event that the pixel which is the object belongs to thecovered background region, sets the mixture ratio α to the estimatedmixture ratio supplied from the estimated mixture ratio processing unit401, and in the event that the pixel which is the object belongs to theuncovered background region, sets the mixture ratio α to the estimatedmixture ratio supplied from the estimated mixture ratio processing unit402, and the processing ends.

[0694] As described above, the mixture ratio calculation unit 104 cancalculate the mixture ratio α which is the amount of featurescorresponding to each pixel based upon the region information suppliedfrom the region specifying unit 103 and the input image.

[0695] The processing of calculation of the mixture ratio α by themixture ratio calculation unit 104 of which configuration is shown inFIG. 70 is the same as the processing described in the flowchart shownin FIG. 71, so description thereof will be omitted.

[0696] The processing for mixture ratio estimation by a modelcorresponding to the covered background region, which corresponds toStep S402 in FIG. 71, will now be described with reference to theflowchart shown in FIG. 72.

[0697] In Step S421, the mixture ratio computation unit 423 obtains thepixel value C of the pixel of interest in the frame #n from the framememory 421.

[0698] In Step S422, the mixture ratio computation unit 423 obtains thepixel value P of the pixel in the frame #n−1, which corresponds to thepixel of interest, from the frame memory 422.

[0699] In Step S423, the mixture ratio computation unit 423 obtains thepixel value N of the pixel in the frame #n+1, which corresponds to thepixel of interest contained in the input image.

[0700] In Step S424, the mixture ratio computation unit 423 computes theestimated mixture ratio based upon the pixel value C of the pixel ofinterest in the frame #n, the pixel value P of the pixel in the frame#n−1, and the pixel value N of the pixel in the frame #n+1.

[0701] In Step S425, the mixture ratio computation unit 423 judgeswhether or not the processing for computation of the estimated mixtureratio has been ended for the entire frame, and in the event thatjudgment is made that the processing for computation of the estimatedmixture ratio has not been ended for the entire frame, the flow returnsto Step S421, and the processing for calculating of the estimatedmixture ratio is repeated for the following pixel.

[0702] In Step S425, in the event that judgment is made that theprocessing for computation of the estimated mixture ratio has been endedfor the entire frame, the processing ends.

[0703] As described above, the estimated mixture ratio processing unit401 can compute the estimated mixture ratio based upon the input image.

[0704] The processing of mixture ratio estimation by a modelcorresponding to the uncovered background region shown in Step S403 inFIG. 71 is the same as the processing indicated in the flowchart shownin FIG. 72, wherein expressions corresponding to a model of theuncovered background region are used, and accordingly descriptionthereof will be omitted.

[0705] Note that the estimated mixture ratio processing unit 442 and theestimated mixture ratio processing unit 443 shown in FIG. 70 compute theestimated mixture ratio by performing the processing the same as theprocessing indicated in the flowchart shown in FIG. 72, and accordinglydescription thereof will be omitted.

[0706] Also, while description has been made with an assumption that theobject corresponding to the background keeps still, the processing forobtaining the mixture ratio α described above may be applied even if theimage corresponding to the background region contains movement. Forexample, in the event that the image corresponding to the backgroundmoves uniformly, the estimated mixture ratio processing unit 401 shiftsthe entire image corresponding to the background movement, and performsprocessing in the same manner as in the case wherein the objectcorresponding to the background keeps still. Also, in the event that theimage corresponding to the background region contains the backgroundmovement which is different at each local position, the estimatedmixture ratio processing unit 401 selects the pixels corresponding tothe background movement as the pixels corresponding to the pixelsbelonging to the mixed region, and performs the processing describedabove.

[0707] Also, an arrangement may be made wherein the mixture ratiocalculation unit 104 performs only the processing of the mixture ratioestimation by a model corresponding to the covered background region forall pixels, and outputs the calculated estimated mixture ratio as themixture ratio α. In this case, the mixture ratio α indicates the ratioof the background components with regard to the pixels belonging to thecovered background region, and indicates the ratio of the foregroundcomponents with regard to the pixels belonging to the uncoveredbackground region. The signal processing device can obtain the mixtureratio α indicating the ratio of the background components with regard tothe pixels belonging to the uncovered background region, by calculatingthe absolute value of the difference between the mixture ratio αcalculated as described above and 1, and setting the calculated absolutevalue for the mixture ratio α, with regard to the pixels belonging tothe uncovered background region.

[0708] Note that similarly, an arrangement may be made wherein themixture ratio calculation unit 104 performs only the processing for themixture ratio estimation by a model corresponding to the uncoveredbackground region for all pixels, and outputs the calculated estimatedmixture ratio as the mixture ratio α.

[0709] Another processing of the mixture ratio calculation unit 104 willnow be described.

[0710] An expression wherein the mixture ratio α and the sum of theforeground components f are approximated spatially can be formed, usingthe nature wherein the mixture ratio α changes linearly corresponding tothe change of the pixel position due to the object corresponding to theforeground moving at a constant velocity within a shutter period. Themixture ratio α is calculated by solving the expression wherein themixture ratio α and the sum of the foreground components f areapproximated, using multiple sets of the pixel value of the pixelbelonging to the mixed region and the pixel value of the pixel belongingto the background region.

[0711] In the event that the change of the mixture ratio α is generallylinear, the mixture ratio α is represented in Expression (23).

α=il+p  (23)

[0712] In Expression (23), i denotes the index in the spatial directionwherein the position of the pixel of interest is 0. l is the inclinationof the straight line of the mixture ratio α. p is the intercept of thestraight line of the mixture ratio α, as well as the mixture ratio α ofthe pixel of interest. In Expression (23), while the index i is known,the inclination l and the intercept p are unknown.

[0713] The correlation between the index i, the inclination l, and theintercept p is shown in FIG. 73.

[0714] In FIG. 73, a white circle indicates the pixel of interest, andsolid circles indicate pixels near the pixel of interest.

[0715] Multiple different mixture ratio α for a plurality of pixels arerepresented by two variables by approximating the mixture ratio α as inExpression (23). In the example shown in FIG. 73, five mixture ratiosfor five pixels are represented by two variables, i.e., the inclinationl and the intercept p.

[0716] In the event of approximating the mixture ratio α in a plannermanner shown in FIG. 74, taking the movement v corresponding to the twodirections of the horizontal direction and the vertical direction of theimage into consideration, the mixture ratio α is represented inExpression (24) by expanding Expression (23) onto a plane. In FIG. 74,the white circle indicates the pixel of interest.

α=jm+kq+p  (24)

[0717] In Expression (24), j is the index in the horizontal directionwherein the position of the pixel of interest is 0, and k is the indexin the vertical direction. m is the inclination of the mixture ratio αin the horizontal direction, and q is the inclination of the plane ofthe mixture ratio α in the vertical direction. p is the intercept of theplane of the mixture ratio α.

[0718] For example, in the frame #n shown in FIG. 64, Expression (25)through Expression (27) hold with regard to C05 through C07,respectively.

C05=α05·B05/v+f05  (25)

C06=α06·B06/v+f06  (26)

C07=α07·B07/v+f07  (27)

[0719] Making an assumption that the foreground components generallyagree, i.e., F01 through F03 are the same, and F01 through F03 arewritten as Fc, Expression (28) holds.

f(x)=(1−α(x))·Fc  (28)

[0720] In Expression (28), x denotes the position in the spatialdirection.

[0721] Rewriting α(x) as Expression (24), Expression (28) may berepresented as Expression (29).

f(x)=(1−(jm+kq+p))·Fc=j·(−m·Fc)+k·(−q·Fc)+((1−p)·Fc)=js+kt+u  (29)

[0722] In Expression (29), (−m·Fc), (−q·Fc), and (1−p)·Fc are rewrittenas Expression (30) through Expression (32).

s=−m·Fc  (30)

t=−q·Fc  (31)

u=(1−p)·Fc  (32)

[0723] In Expression (29), j is the index in the horizontal directionwherein the position of pixel of interest is 0, and k is the index inthe vertical direction.

[0724] As described above, since an assumption is made that the objectcorresponding to the foreground moves at a constant velocity within ashutter period, and the components corresponding to the foregroundgenerally agree, the sum of the foreground components is approximated inExpression (29).

[0725] Note that in the event of approximating the mixture ratio αlinearly, the sum of the foreground components may be represented inExpression (33).

f(x)=is+u  (33)

[0726] Rewriting the mixture ratio α and the sum of the foregroundcomponents in Expression (13) using Expression (24) and Expression (29),the pixel value M is represented in Expression (34).

M=(jm+kq+p)·B+js+kt+u=jB·m+kB·q+B·p+j·s+k·t+u  (34)

[0727] In Expression (34), the unknown variables are the six values ofthe inclination of the plane of the mixture ratio α in the horizontaldirection, m, the inclination of the plane of the mixture ratio α in thevertical direction, q, the intercepts of the plane of the mixture ratioα, p, s, t, and u.

[0728] Setting the pixel value M and pixel value B for the normalequation represented in Expression (34) corresponding to the pixels nearthe pixel of interest, the mixture ratio α is calculated by solvingmultiple normal equations wherein the pixel value M and the pixel valueB have been set, by the least square method.

[0729] For example, with the index j of the pixel of interest in thehorizontal direction as 0, with the index k of the pixel of interest inthe vertical direction as 0, and setting the pixel value M or the pixelvalue B for the normal equation represented in Expression (34) withregard to 3×3 pixels near the pixel of interest, Expression (35) throughExpression (43) are obtained.

M _(−1,−1)=(−1)·B _(−1,−1) ·m+(−1)·B _(−1,−1) ·q+B _(−1,−1)·p+(−1)·s+(−1)·t+u  (35)

M _(0,−1)=(0)·B _(0,−1) ·m+(−1)·B _(0,−1) ·q+B _(0,−1)·p+(0)·s+(−1)·t+u  (36)

M _(+1,−1)=(+1)·B _(+1,−1) ·m+(−1)·B _(+1,−1) ·q+B _(+1,−1)·p+(+1)·s+(−1)·t+u  (37)

M _(−1,0)=(−1)·B _(−1,0) ·m+(0)·B _(−1,0) ·q+B _(−1,0)·p+(−1)·s+(0)·t+u  (38)

M _(0,0)=(0)·B _(0,0) ·m+(0)·B _(0,0) ·q+B _(0,0) ·p+(0)·s+(0)·t+u  (39)

M _(+1,0)=(+1)·B _(+1,0) ·m+(0)·B _(+1,0) ·q+B _(+1,0)·p+(+1)·s+(0)·t+u  (40)

M _(−1,+1)=(−1)·B _(−1,+1) ·m+(+1)·B _(−1,+1) ·q+B _(−1,+1)·p+(−1)·s+(+1)·t+u  (41)

M _(0,+1)=(0)·B _(0,+1) ·m+(+1)·B _(0,+1) ·q+B _(0,+1)·p+(0)·s+(+1)·t+u  (42)

M _(+1,+1)=(+1)·B _(+1,+1) ·m+(+1)·B _(+1,+1) ·q+B _(+1,+1)·p+(+1)·s+(+1)·t+u  (43)

[0730] Since the index of the pixel of interest in the horizontaldirection, j, is 0, and the index in the vertical direction, k, is 0,the mixture ratio α of the pixel of interest is equal to the valuewherein j=0 and k=0, from Expression (24), i.e., the intercept p.

[0731] Accordingly, the intercept p can be output as the mixture ratio αby calculating the inclination in the horizontal direction, m, theinclination in the vertical direction, q, the intercept p, s, t, and u,by the least square method, based upon the nine expressions ofExpression (35) through Expression (43).

[0732] More specific procedures for calculating the mixture ratio α byapplying the least square method will now be described.

[0733] Representing the index i and the index k with one index x, therelationship between the index i, the index k, and the index x, isrepresented in Expression (44).

x=(j+1)·3+(k+1)  (44)

[0734] The inclination in the horizontal direction, m, the inclinationin the vertical direction, q, the intercept p, s, t, and u, arerepresented by variables, w0, w1, w2, w3, w4, and w5, respectively, andjB, kB, B, j, k, and l are represented by a0, a1, a2, a3, a4, and a5,respectively. Taking the margin of error ex into consideration,Expression (35) through Expression (43) are rewritten as Expression(45). $\begin{matrix}{M_{x} = {{\sum\limits_{y = 0}^{5}{a_{y} \cdot w_{y}}} + e_{x}}} & (45)\end{matrix}$

[0735] In Expression (45), x denotes one of the integers between 0 and8.

[0736] Expression (46) may be derived from Expression (45).$\begin{matrix}{e_{x} = {M_{x} - {\sum\limits_{y = 0}^{5}{a_{y} \cdot w_{y}}}}} & (46)\end{matrix}$

[0737] To apply the least square method, the sum of squares of margin oferror E is defined as represented in Expression (47). $\begin{matrix}{E = {\sum\limits_{x = 0}^{8}e_{x}^{2}}} & (47)\end{matrix}$

[0738] To minimize the margin of error, the partial derivative of thesquared-sum of the margin of error E from the variable Wv should be 0.Here v is one of the integers between 0 through 5. Accordingly, wy iscalculated so as to satisfy Expression (48). $\begin{matrix}\begin{matrix}{\frac{\partial E}{\partial w_{v}} = {2 \cdot {\sum\limits_{x = 0}^{8}{e_{x} \cdot \frac{\partial e_{x}}{\partial w_{v}}}}}} \\{= {{2 \cdot {\sum\limits_{x = 0}^{8}{e_{x} \cdot a_{v}}}} = 0}}\end{matrix} & (48)\end{matrix}$

[0739] Substituting Expression (46) for Expression (48), Expression (49)is obtained. $\begin{matrix}{{\sum\limits_{x = 0}^{8}\left( {a_{v} \cdot {\sum\limits_{y = 0}^{5}{a_{y} \cdot w_{y}}}} \right)} = {\sum\limits_{x = 0}^{8}{a_{v} \cdot M_{x}}}} & (49)\end{matrix}$

[0740] Applying, for example, the sweeping method (Gauss-Jordanelimination) and so forth, to the six expressions each of which isobtained by substituting one of the integers between 0 and 5 for v inExpression (49), wy is calculated. As described above, w0 denotes theinclination in the horizontal direction, m, w1 denotes the inclinationin the vertical direction, q, w2 denotes the intercept p, w3 denotes s,w4 denotes t, and w5 denotes u.

[0741] As described above, the inclination in the horizontal direction,m, the inclination in the vertical direction, q, the intercept p, s, t,and u may be obtained by applying the least square method to anexpression wherein the pixel value M and the pixel value B have beenset.

[0742] In the description corresponding to Expression (35) throughExpression (43), while description has been made with the pixel value ofthe pixel included in the mixed region as M, and the pixel value of thepixel included in the background region as B, the normal equation needsto be formed for each case of the pixel of interest being included inthe covered background region, and being included in the uncoveredbackground region.

[0743] For example, in a case of obtaining the mixture ratio α of thepixel included in the covered background region of the frame #n shown inFIG. 64, the pixels of the frame #n, C04 through C08, and the pixelvalues of the pixels of the frame #n−1, P04 through P08, are set for thenormal equation.

[0744] In a case of obtaining the mixture ratio α of the pixel includedin the uncovered background region of the frame #n shown in FIG. 65, thepixels of the frame #n, C28 through C32, and the pixel values of thepixels of the frame #n+1, N28 through N32, are set for the normalequation.

[0745] Also, for example, in the event of calculating the mixture ratioα of the pixel included in the covered background region shown in FIG.75, Expression (50) through Expression (58) described below may beformed. The pixel value of the pixel for calculation of the mixtureratio α is Mc5. In FIG. 75, white circles indicate the pixels which areregarded as backgrounds, and solid circles indicate the pixels which areregarded as pixels in the mixed region.

Mc1=(−1)·Bc1·m+(−1)·Bc1·q+Bc1·p+(−1)·s+(−1)·t+u  (50)

Mc2=(0)·Bc2·m+(−1)·Bc2·q+Bc2·p+(0)·s+(−1)·t+u  (51)

Mc3=(+1)·Bc3·m+(−1)·Bc3·q+Bc3·p+(+1)·s+(−1)·t+u  (52)

Mc4=(−1)·Bc4·m+(0)·Bc4·q+Bc4·p+(−1)·s+(0)·t+u  (53)

Mc5=(0)·Bc5·m+(0)·Bc5·q+Bc5·p+(0)·s+(0)·t+u  (54)

Mc6=(+1)·Bc6·m+(0)·Bc6·q+Bc6·p+(+1)·s+(0)·t+u  (55)

Mc7=(−1)·Bc7·m+(+1)·Bc7·q+Bc7·p+(−1)·s+(+1)·t+u  (56)

Mc8=(0)·Bc8·m+(+1)·Bc8·q+Bc8·p+(0)·s+(+1)·t+u  (57)

Mc9=(+1)·Bc9·m+(+1)·Bc9·q+Bc9·p+(+1)·s+(+1)·t+u  (58)

[0746] In the event of calculating the mixture ratio α of the pixelincluded in the covered background region in the frame #n, the pixelvalues Bc1 through Bc9 of the pixels in the background region in theframe #n−1 corresponding to the pixels in the frame #n, are used inExpression (50) through Expression (58).

[0747] In the event of calculating the mixture ratio α of the pixelincluded in the uncovered background region shown in FIG. 75, Expression(59) through Expression (67) described below may be formed. The pixelvalue of the pixel for calculation of the mixture ratio α is Mu5.

Mu1=(−1)·Bu1·m+(−1)·Bu1·q+Bu1·p+(−1)·s+(−1)·t+u  (59)

Mu2=(0)·Bu2·m+(−1)·Bu2·q+Bu2·p+(0)·s+(−1)·t+u  (60)

Mu3=(+1)·Bu3·m+(−1)·Bu3·q+Bu3·p+(+1)·s+(−1)·t+u  (61)

Mu4=(−1)·Bu4·m+(0)·Bu4·q+Bu4·p+(−1)·s+(0)·t+u  (62)

Mu5=(0)·Bu5·m+(0)·Bu5·q+Bu5·p+(0)·s+(0)·t+u  (63)

Mu6=(+1)·Bu6·m+(0)·Bu6·q+Bu6·p+(+1)·s+(0)·t+u  (64)

Mu7=(−1)·Bu7·m+(+1)·Bu7·q+Bu7·p+(−1)·s+(+1)·t+u  (65)

Mu8=(0)·Bu8·m+(+1)·Bu8·q+Bu8·p+(0)·s+(+1)·t+u  (66)

Mu9=(+1)·Bu9·m+(+1)·Bu9·q+Bu9·p+(+1)·s+(+1)·t+u  (67)

[0748] In the event of calculating the mixture ratio α of the pixelincluded in the uncovered background region in the frame #n, the pixelvalues Bu1 through Bu9 of the pixels in the background region in theframe #n+1 corresponding to the pixels in the frame #n, are used inExpression (59) through Expression (67).

[0749]FIG. 76 is a block diagram which illustrates the configuration ofthe estimated mixture ratio processing unit 401. The image input to theestimated mixture ratio processing unit 401 is supplied to a delaycircuit 501 and an addition unit 502.

[0750] The delay circuit 501 delays the input image by one frame, andsupplies to the addition unit 502. At the point that the frame #n isinput to the addition unit 502 as an input image, the delay circuit 501supplies the frame #n−1 to the addition unit 502.

[0751] The addition unit 502 sets the pixel values of the pixels nearthe pixel for calculation of the mixture ratio α, and the pixel valuesof the frame #n−1, for the normal equation. For example, the additionunit 502 sets the pixel values Mc1 through Mc9, and the pixel values Bc1through Bc9 for the normal equation based upon Expression (50) throughExpression (58). The addition unit 502 supplies the normal equation forwhich the pixel values have been set, to a computation unit 503.

[0752] The computation unit 503 obtains the estimated mixture ratio bysolving the normal equation supplied from the addition unit 502 by thesweeping method or the like, and outputs the obtained estimated mixtureratio.

[0753] As described above, the estimated mixture ratio processing unit401 can calculate the estimated mixture ratio based upon the inputimage, and supply to the mixture ratio decision unit 403.

[0754] Note that the estimated mixture ratio processing unit 402 has thesame configuration as the estimated mixture ratio processing unit 401,and accordingly description thereof will be omitted.

[0755]FIG. 77 is a diagram which illustrates an example of the estimatedmixture ratio calculated by the estimated mixture ratio processing unit401. FIG. 77 illustrates the estimated mixture ratio with regard to oneline, wherein the movement v of the foreground corresponding to theobject which moves at a constant velocity is 11, and the results arecalculated by the expression generated in increments of blocks 7×7pixels.

[0756] It can be understood that the estimated mixture ratio changesgenerally linearly in the mixed region as shown in FIG. 63.

[0757] The mixture ratio decision unit 403 sets the mixture ratio basedupon the region information indicating which of the foreground region,the background region, the covered background region, or the uncoveredbackground region the pixel for calculation of the mixture ratio belongsto, supplied from the region specifying unit 101. In the event that thepixel which is the object belongs to the foreground region, the mixtureratio decision unit 403 sets the mixture ratio to 0, in the event thatthe pixel which is the object belongs to the background region, sets themixture ratio to 1, in the event that the pixel which is the objectbelongs to the covered background region, sets the mixture ratio to theestimated mixture ratio supplied from the estimated mixture ratioprocessing unit 401, and in the event that the pixel which is the objectbelongs to the uncovered background region, sets the mixture ratio tothe estimated mixture ratio supplied from the estimated mixture ratioprocessing unit 402. The mixture ratio decision unit 403 outputs themixture ratio which is set based upon the region information.

[0758] Referring to the flowchart shown in FIG. 78, the processing forcalculation of the mixture ratio by the mixture ratio calculation unit102 in a case that the estimated mixture ratio processing unit 401 has aconfiguration shown in FIG. 76 will be described. In Step S501, themixture ratio calculation unit 102 obtains the region informationsupplied from the region specifying unit 101. In Step S502, theestimated mixture ratio processing unit 401 performs the processing ofmixture ratio estimation by a model corresponding to the coveredbackground region, and supplies the estimated mixture ratio to themixture ratio decision unit 403. Details of the processing of mixtureratio estimation will be described later with reference to the flowchartshown in FIG. 79.

[0759] In Step S503, the estimated mixture ratio processing unit 402performs the processing of mixture ratio estimation by a modelcorresponding to the uncovered background region, and supplies theestimated mixture ratio to the mixture ratio decision unit 403.

[0760] In Step S504, the mixture ratio calculation unit 102 judgeswhether or not the mixture ratio has been estimated for the entireframe, and in the event that judgment is made that the mixture ratio hasnot been estimated for the entire frame, the flow returns to Step S502,and the processing of mixture ratio estimation for the following pixelis performed.

[0761] In Step S504, in the event that judgment is made that the mixtureratio has been estimated for the entire frame, the flow proceeds to StepS505, and the mixture ratio decision unit 403 sets the mixture ratiobased upon the region information which indicates which of theforeground region, the background region, the covered background region,or the uncovered background region the pixel of calculation of themixture ratio belongs to, supplied from the region specifying unit 101.In the event that the pixel which is the object belongs to theforeground region, the mixture ratio decision unit 403 sets the mixtureratio to 0, in the event that the pixel which is the object belongs tothe background region, sets the mixture ratio to 1, in the event thatthe pixel which is the object belongs to the covered background region,sets the mixture ratio to the estimated mixture ratio supplied from theestimated mixture ratio processing unit 401, and in the event that thepixel which is the object belongs to the uncovered background region,sets the mixture ratio to the estimated mixture ratio supplied from theestimated mixture ratio processing unit 402, and processing ends.

[0762] As described above, the mixture ratio calculation unit 102 cancalculate the mixture ratio α which is the amount of featurescorresponding to each pixel based upon the region information suppliedfrom the region specifying unit 101 and the input image.

[0763] Using the mixture ratio α enables the separation of theforeground components and the background components contained in thepixel value while leaving the information of movement blurring containedin the image corresponding to the moving object.

[0764] Also, synthesizing an image based upon the mixture ratio αenables creation of an image containing accurate movement blurringcorresponding to the speed of the object which moves as if image takingof the real world had been performed again.

[0765] The processing of the mixture ratio estimation by a modelcorresponding to the covered background region, which corresponds toStep S502 shown in FIG. 78, will now be described with reference to theflowchart shown in FIG. 79.

[0766] In Step S521, the addition unit 502 sets the pixel valuescontained in the input image and the pixels contained the image suppliedfrom the delay circuit 501 for the normal equation corresponding to amodel of the covered background region.

[0767] In Step S522, the estimated mixture ratio processing unit 401judges whether or not setting of the pixel which is the object hasended, and in the event that judgment is made that the setting for thepixel which is the object has not ended, the flow returns to Step S521,and the processing of setting of the pixel value for the normal equationis repeated.

[0768] In Step S522, in the event that judgment is made that setting ofpixel values with regard to the pixel which is the object has ended, theflow proceeds to Step S523, and the computation unit 503 computes theestimated mixture ratio based upon the normal equation which the pixelvalues have been set for, and outputs the obtained estimated mixtureratio.

[0769] As described above, the estimated mixture ratio processing unit401 can computes the estimated mixture ratio based upon the input image.

[0770] The processing of mixture ratio estimation by a modelcorresponding to the uncovered background region in Step S153 shown inFIG. 78 is the same as the processing shown in the flowchart in FIG. 79using the normal equation corresponding to a model of the uncoveredbackground region, and accordingly, description thereof will be omitted.

[0771] Note that while description has been made with an assumption thatthe object corresponding to the background keeps still, the processingof obtaining the mixture ratio described above may be applied even ifthe image corresponding to the background contains movement. Forexample, in the event that the image corresponding to the backgroundregion moves uniformly, the estimated mixture ratio processing unit 401shifts the entire image corresponding to the movement, and performsprocessing in the same manner as a case wherein the object correspondingto the background keeps still. Also, in the event that the imagecorresponding to the background contains different movement at eachlocal position, the estimated mixture ratio processing unit 401 selectspixels corresponding to the movement as pixels corresponding to pixelsbelonging to the mixed region, and performs the above-describedprocessing.

[0772] The foreground/background separation unit 105 will now bedescribed. FIG. 80 is a block diagram which illustrates an example ofthe configuration of the foreground/background separation unit 105. Theinput image supplied to the foreground/background separation unit 105 issupplied to a separation unit 601, a switch 602, and a switch 604. Theregion information indicating the covered background region and theuncovered background region, which is supplied from the regionspecifying unit 103, is supplied to the separation unit 601. The regioninformation indicating the foreground region is supplied to the switch602. The region information indicating the background region is suppliedto the switch 604.

[0773] The mixture ratio α supplied from the mixture ratio calculationunit 104 is supplied to the separation unit 601.

[0774] The separation unit 601 separates the foreground components fromthe input image based upon the region information indicating the coveredbackground region, the region information indicating the uncoveredbackground region, and the mixture ratio α, and supplies the separatedforeground components to a synthesizing unit 603, as well as separatingthe background components from the input image, and supplying theseparated background components to the synthesizing unit 605.

[0775] In the event that the pixel corresponding to the foreground isinput, the switch 602 is closed based upon the region informationindicating the foreground region, and supplies only the pixelscorresponding to the foreground included in the input image to thesynthesizing unit 603.

[0776] In the event that the pixel corresponding to the background isinput, the switch 604 is closed based upon the region informationindicating the background region, and supplies only the pixelscorresponding to the background included in the input image to thesynthesizing unit 605.

[0777] The synthesizing unit 603 synthesizes the foreground componentimage based upon the components corresponding to the foreground suppliedfrom the separation unit 601, and the pixels corresponding to theforeground supplied from the switch 602, and outputs the synthesizedforeground component image. Since the foreground region and the mixedregion are not overlapped, the synthesizing unit 603 synthesizes theforeground component image, for example, by applying the logical sumcomputation to the components corresponding to the foreground, and thepixels corresponding to the foreground.

[0778] In the initialization processing which is performed in the firststage of the foreground component image synthesizing processing, thesynthesizing unit 603 stores the image, wherein all the pixel values are0, in built-in frame memory, and in the foreground component imagesynthesizing processing, the synthesizing unit 603 stores (oroverwrites) the foreground component image. Accordingly, the pixelcorresponding to the background region, which is the foregroundcomponent image output from the synthesizing unit 603, stores 0 as apixel value.

[0779] The synthesizing unit 605 synthesizes the background componentimage based upon the components corresponding to the background suppliedfrom the separation unit 601, and the pixels corresponding to thebackground supplied from the switch 604, and outputs the synthesizedbackground component image. Since the background region and the mixedregion are not overlapped, the synthesizing unit 605 synthesizes thebackground component image, for example, by applying the logical sumcomputation to the components corresponding to the background, and thepixels corresponding to the background.

[0780] In the initialization processing which is performed in the firststage of the background component image synthesizing processing, thesynthesizing unit 605 stores the image, wherein all the pixel values are0, in built-in frame memory, and in the background component imagesynthesizing processing, the synthesizing unit 605 stores (oroverwrites) the background component image. Accordingly, the pixelcorresponding to the foreground region, which is the backgroundcomponent image output from the synthesizing unit 605, stores 0 as apixel value.

[0781]FIG. 81A is a diagram which illustrates the input image input tothe foreground/background separation unit 105, and the foregroundcomponent image and the background component image output from theforeground/background separation unit 105. FIG. 81B is a diagram whichillustrates a model corresponding to the input image input to theforeground/background separation unit 105, and the foreground componentimage and the background component image output from theforeground/background separation unit 105.

[0782]FIG. 81A is a schematic diagram which illustrates the displayedimage, and FIG. 81B is a model diagram wherein one line of pixelsincluding pixels belonging to the foreground region, pixels belonging tothe background region, and pixels belonging to the mixed region,corresponding to FIG. 81A, develop over the time direction.

[0783] As shown in FIG. 81A and FIG. 81B, the background component imageoutput from the foreground/background separation unit 105 is made up ofpixels belonging to the background region and background componentscontaining pixels in the mixed region.

[0784] As shown in FIG. 81A and FIG. 81B, the foreground component imageoutput from the foreground/background separation unit 105 is made up ofpixels belonging to the foreground region and foreground componentscontained in pixels in the mixed region.

[0785] The pixel value of the pixel in the mixed region is separatedinto the background components and the foreground components by theforeground/background separation unit 105. The separated backgroundcomponents make up a background component image along with pixelsbelonging to the background region. The separated foreground componentsmake up a foreground component image along with pixels belonging to theforeground region.

[0786] As described above, in the foreground component image, the pixelvalues of the pixels corresponding to the background region are set to0, and the pixels corresponding to the foreground region and the pixelscorresponding to the mixed region are set to valid pixel values.Similarly, in the background component image, the pixel values of thepixels corresponding to the foreground region are set to 0, and thepixels corresponding to the background region and the pixelscorresponding to the mixed region are set to valid pixel values.

[0787] A description will now be made regarding the separationprocessing of the foreground components and the background componentsfrom the pixel belonging to the mixed region performed by the separationunit 601.

[0788]FIG. 82 is a model of an image which indicates two frames of theforeground components and the background components, including theforeground corresponding to the object which moves from the left to theright in the drawing. In the model of the image shown in FIG. 82, themovement amount v of the foreground is 4, and the virtual dividingnumber is 4.

[0789] In the frame #n, the left-most pixel and the fourteenth througheighteenth pixels from the left are made up of only the backgroundcomponents, and belong to the background region. In the frame #n, thesecond through fourth pixels from the left are made up of the backgroundcomponents and the foreground components, and belong to the uncoveredbackground region. In the frame #n, the eleventh through thirteenthpixels from the left are made up of the background components and theforeground components, and belong to the covered background region. Inthe frame #n, the fifth through tenth pixels from the left are made upof only the foreground components, and belong to the foreground region.

[0790] In the frame #n+1, the first through fifth pixels from the leftand the eighteenth pixel from the left are made up of only thebackground components, and belong to the background region. In the frame#n+1, the sixth through eighth pixels from the left contain thebackground components and the foreground components, and belong to theuncovered background region. In the frame #n+1, the fifteenth throughseventeenth pixels from the left contain the background components andthe foreground components, and belong to the covered background region.In the frame #n+1, the ninth through fourteen pixels from the left aremade up of only the foreground components, and belong to the foregroundregion.

[0791]FIG. 83 is a diagram which describes the processing for separationof the foreground components from the pixel belonging to the coveredbackground region. In FIG. 83, α1 through α18 are the mixture ratioscorresponding to the pixels in the frame #n, respectively. In FIG. 83,the fifteenth through seventeenth pixels from the left belongs to thecovered background region.

[0792] The pixel value C15 of the fifteenth pixel from the left in theframe #n is represented in Expression (68).

C15=B15/v+F09/v+F08/v+F07/v=α15·B15+F09/v+F08/v+F07/v=α15·P15+F09/v+F08/v+F07/v  (68)

[0793] Here, α15 denotes the mixture ratio of the fifteenth pixel fromthe left in the frame #n. P15 denotes the pixel value of the fifteenthpixel from the left in the frame #n−1.

[0794] The sum f15 of the foreground components of the fifteenth pixelfrom the left in the frame #n is represented in Expression (69) basedupon Expression (68).

f15=F09/v+F08/v+F07/v=C15−α15·P15  (69)

[0795] Similarly, the sum f16 of the foreground components of thesixteenth pixel from the left in the frame #n is represented inExpression (70), and the sum f17 of the foreground components of theseventeenth pixel from the left in the frame #n is represented inExpression (71).

f16=C16−α16·P16  (70)

f17=C17−α17·P17  (71)

[0796] As described above, the foreground component fc contained in thepixel value C of the pixel belonging to the covered background region iscalculated by Expression (72).

fc=C−α·P  (72)

[0797] P denotes the pixel value of the corresponding pixel in theprevious frame.

[0798]FIG. 84 is a diagram which describes the processing for separatingthe foreground components from the pixel belonging to the uncoveredbackground region. In FIG. 84, α1 through α18 denote the mixture ratiocorresponding to the pixels in the frame #n, respectively. In FIG. 84,the second through fourth pixels from the left belong to the uncoveredbackground region.

[0799] The pixel value C02 of the second pixel from the left in theframe #n is represented in Expression (73).

C02=B02/v+B02/v+B02/v+F01/v=α2·B02+F01/v=α2·N02+F01/v  (73)

[0800] Here, α2 denotes the mixture ratio of the second pixel from theleft in the frame #n. N02 denotes the pixel value of the second pixelfrom the left in the frame #n+1.

[0801] The foreground component sum of the second pixel from the left inthe frame #n, f02, is represented in Expression (74) based uponExpression (73).

f02=F01/v=C02−α2·N02  (74)

[0802] Similarly, the foreground component sum of the third pixel fromthe left in the frame #n, f03, is represented in Expression (75), andthe foreground component sum of the fourth pixel from the left in theframe #n, f04, is represented in Expression (76).

f03=C03−α3·N03  (75)

f04=C04−α4·N04  (76)

[0803] As described above, the foreground component fu contained in thepixel value C of the pixel belonging to the uncovered background regionis calculated by Expression (77).

fu=C−α·N  (77)

[0804] N denotes the pixel value of the corresponding pixel in thefollowing frame.

[0805] As described above, the separation unit 601 can separate theforeground components and the background components from the pixelbelonging to the mixed region based upon the information indicating thecovered background region and the information indicating the uncoveredbackground region, which is included in the region information, and themixture ratio α for each pixel.

[0806]FIG. 85 is a block diagram which illustrates an example of theconfiguration of the separation unit 601 for performing the processingdescribed above. The image input to the separation unit 601 is suppliedto frame memory 621, and the region information indicating the coveredbackground region and the uncovered background region supplied from themixture ratio calculation unit 104, and the mixture ratio α is input toa separation processing block 622.

[0807] The frame memory 621 stores the input image in increments offrames. In the event that the object of processing is the frame #n, theframe memory 621 stores the frame #n−1 which is the frame previous tothe frame #n, frame #n, and the frame #n+1 which is the frame followingthe frame #n.

[0808] The frame memory 621 supplies the corresponding pixels in theframe #n−1, the frame #n, and the frame #n+1 to the separationprocessing block 622.

[0809] The separation processing block 622 separates the foregroundcomponents and the background components from the pixel belonging to themixed region in the frame #n by applying the computation described withreference to FIG. 83 and FIG. 84 to the pixel values of correspondingpixels in the frame #n−1, the frame #n, and the frame #n+1, suppliedfrom the frame memory 621, based upon the region information indicatingthe covered background region and the uncovered background region, andthe mixture ratio α, and supplies to the frame memory 623.

[0810] The separation processing block 622 comprises an uncovered regionprocessing unit 631, a covered region processing unit 632, asynthesizing unit 633, and a synthesizing unit 634.

[0811] A multiplication device 641 of the uncovered region processingunit 631 multiplies the pixel value of the pixel of the frame #n+1supplied from the frame memory 621 by the mixture ratio α, and outputsto a switch 642. In the event that the pixel in the frame #n suppliedfrom the frame memory 621 (which is corresponding to the pixel of theframe #n+1) belongs to the uncovered background region, the switch 642is closed, the pixel value which is multiplied by the mixture ratio αsupplied from the multiplication device 641 is supplied to a computingdevice 643 and the synthesizing unit 634. The value wherein the pixelvalue of the pixel of the frame #n+1 output from the switch 642 ismultiplied by the mixture ratio α is the same as the backgroundcomponent of the pixel value of the corresponding pixel in the frame #n.

[0812] The computing device 643 obtains the foreground components bysubtracting the background components supplied from the switch 642 fromthe pixel value of the pixel of the frame #n supplied from the framememory 621. The computing device 643 supplies the foreground componentsof the pixel in the frame #n belonging to the uncovered backgroundregion, to the synthesizing unit 633.

[0813] A multiplication device 651 of the covered region processing unit632 multiplies the pixel value of the pixel of the frame #n−1 suppliedfrom the frame memory 621 by the mixture ratio α, and outputs to aswitch 652. In the event that the pixel in the frame #n supplied fromthe frame memory 621 (corresponding to the pixel of the frame #n−1)belongs to the covered background region, the switch 652 is closed, andthe pixel value which is multiplied by the mixture ratio α supplied fromthe multiplication device 651 is supplied to a computing device 653 andthe synthesizing unit 634. The value wherein the pixel value of thepixel of the frame #n−1 output from the switch 652 is multiplied by themixture ratio α, is the same as the background component of the pixelvalue of the corresponding pixel in the frame #n.

[0814] The computing device 653 obtains the foreground components bysubtracting the background components supplied from the switch 652 fromthe pixel value of the pixel of the frame #n supplied from the framememory 621. The computing device 653 supplies the foreground componentsof the pixel in the frame #n belonging to the covered background region,to the synthesizing unit 633.

[0815] The synthesizing unit 633 synthesizes the foreground componentsof the pixel belonging to the uncovered background region in the frame#n supplied from the computing device 643, and the foreground componentsof the pixel belonging to the covered background region supplied fromthe computing device 653, and supplies to the frame memory 623.

[0816] The synthesizing unit 634 synthesizes the background componentsof the pixel belonging to the uncovered background region in the frame#n supplied from the switch 642, and the background components of thepixel belonging to the covered background region supplied from theswitch 652, and supplies to the frame memory 623.

[0817] The frame memory 623 stores the foreground components and thebackground components of the pixels in the mixed region in the frame #n,supplied from the separation processing block 622, respectively.

[0818] The frame memory 623 outputs the foreground components of thepixels in the mixed region in the frame #n stored therein, and thebackground components of the pixels in the mixed region in the frame #nstored therein.

[0819] Using the mixture ratio α which is the feature amount enablescomplete separation of the foreground components and the backgroundcomponents, contained in the pixel value.

[0820] The synthesizing unit 603 generates the foreground componentimage by synthesizing the foreground components of the pixel in themixed region in the frame #n output from the separation unit 601, andthe pixels belonging to the foreground region. The synthesizing unit 605generates the background component image by synthesizing the backgroundcomponents of the pixels in the mixed region in the frame #n output fromthe separation unit 601, and pixels belonging to the background region.

[0821]FIG. 86A is a diagram which illustrates an example of theforeground component image corresponding to the frame #n shown in FIG.82. FIG. 86B is a diagram which illustrates an example of the backgroundcomponent image corresponding to the frame #n shown in FIG. 82.

[0822]FIG. 86A illustrates an example of the foreground component imagecorresponding to the frame #n shown in FIG. 82. Since the left-mostpixel and the fourteenth pixel from the left are made up of only thebackground components before separation of the foreground and thebackground, the pixel values are 0.

[0823] The second through fourth pixels from the left belong to theuncovered background region prior to the foreground and the backgroundbeing separated, with the background components being 0, and theforeground components being left as they are. The eleventh throughthirteenth pixels belong to the covered background region beforeseparation of the foreground and the background, and the backgroundcomponents are 0, and the foreground components are left as they are.Since the fifth through tenth pixels from the left are made up of onlythe foreground components, those are left as they are.

[0824]FIG. 86B illustrates an example of the background component imagecorresponding to the frame #n shown in FIG. 82. The left-most pixel andthe fourteenth pixel from the left are made up of only the backgroundcomponents prior to the foreground and the background being separated,and accordingly, those are left as they are.

[0825] The second through fourth pixels from the left belong to theuncovered background region prior to the foreground and the backgroundbeing separated, with the foreground components being 0, and thebackground components being left as they are. The eleventh throughthirteenth pixels belong to the covered background region prior to theforeground and the background being separated, the foreground componentsbeing 0, and the background components being left as they are. The fifththrough tenth pixels from the left are made up of only the foregroundcomponents prior to the foreground and the background being separated,and accordingly the pixel values are 0.

[0826] The separation processing for the foreground and the backgroundby the foreground/background separation unit 105 will now be described,with reference to the flowchart shown in FIG. 87. In Step S601, theframe memory 621 of the separation unit 601 obtains the input image, andstores the frame #n which is the object of the separation of theforeground and the background, as well as the previous frame #n−1 andthe following frame #n+1.

[0827] In Step S602, the separation processing block 622 of theseparation unit 601 obtains the region information supplied from themixture ratio calculation unit 104. In Step S603, the separationprocessing block 622 of the separation unit 601 obtains the mixtureratio α supplied from the mixture ratio calculation unit 104.

[0828] In Step S604, the uncovered region processing unit 631 extractsthe background components from the pixel value of the pixel belonging tothe uncovered background region supplied from the frame memory 621 basedupon the region information and the mixture ratio α.

[0829] In Step S605, the uncovered region processing unit 631 extractsthe foreground components from the pixel value of the pixel belonging tothe uncovered background region supplied from the frame memory 621 basedupon the region information and the mixture ratio α.

[0830] In Step S606, the covered region processing unit 632 extracts thebackground components from the pixel value of the pixel belonging to thecovered background region supplied from the frame memory 621 based uponthe region information and the mixture ratio α.

[0831] In Step S607, the covered region processing unit 632 extracts theforeground components from the pixel value of the pixel belonging to thecovered background region supplied from the frame memory 621 based uponthe region information and the mixture ratio α.

[0832] In Step S608, the synthesizing unit 633 synthesizes theforeground components of the pixel belonging to the uncovered backgroundregion extracted in the processing in Step S605, and the foregroundcomponents of the pixel belonging to the covered background regionextracted in the processing in Step S607. The synthesized foregroundcomponents are supplied to the synthesizing unit 603. Moreover, thesynthesizing unit 603 synthesizes the pixels belonging to the foregroundregion supplied via the switch 602, and the foreground componentssupplied from the separation unit 601, and generates the foregroundcomponent image.

[0833] In Step S609, the synthesizing unit 634 synthesizes thebackground components of the pixel belonging to the uncovered backgroundregion extracted in the processing in Step S604, and the backgroundcomponents of the pixel belonging to the covered background regionextracted in the processing in Step S606. The synthesized backgroundcomponents are supplied to the synthesizing unit 605. Moreover, thesynthesizing unit 605 synthesizes the pixels belonging to the backgroundregion supplied via the switch 604, and the background componentssupplied from the separation unit 601, and generates the backgroundcomponent image.

[0834] In Step S610, the synthesizing unit 603 outputs the foregroundcomponent image. In Step S611, the synthesizing unit 605 outputs thebackground component image, and the processing ends.

[0835] As described above, the foreground/background separation unit 105can separate the foreground components and the background componentsfrom the input image based upon the region information and the mixtureratio α, and output the foreground component image which is made up ofonly the foreground components, and the background component image whichis made up of only the background components.

[0836] The removal of movement blurring from the foreground componentimage will now be described.

[0837]FIG. 88 is a block diagram which illustrates an example of theconfiguration of the movement blurring removal unit 106. The movementvector and the position information thereof supplied from the movementdetecting unit 102, and the region information supplied from the regionspecifying unit 103 are supplied to a processing increment decision unit801 and the modeling unit 802. The foreground component image suppliedfrom the foreground/background separation unit 105 is supplied to theaddition unit 804.

[0838] The processing increment decision unit 801 supplies theprocessing increment generated based upon the movement vector, theposition information thereof, and the region information, as well as themovement vector, to the modeling unit 802. The processing incrementdecision unit 801 supplies the generated processing increment to theaddition unit 804.

[0839] The processing increment generated by the processing incrementdecision unit 801 denoted by A in FIG. 89, as illustrated by an examplein FIG. 89, indicates the pixels arrayed sequentially in a movementdirection beginning at the pixel corresponding to the covered backgroundregion of the foreground component image up to the pixel correspondingto the uncovered background region, or the pixels arrayed sequentiallyin a movement direction beginning at the pixel corresponding to theuncovered background region up to the pixel corresponding to the coveredbackground region. The processing increment is made up of, for example,two pieces of data of the upper-left point (the left-most or the topmostposition of the pixel, which is the pixel designated by the processingincrement) and the bottom-right point.

[0840] The modeling unit 802 performs modeling based upon the movementvector and the input processing increment. More specifically, forexample, an arrangement may be made wherein the modeling unit 802 storesthe number of pixels included in the processing increment, the virtualdividing number of the pixel value in the time direction, and multiplemodels corresponding to the number of the foreground components for eachpixel beforehand, and selects a model which designates thecorrespondence of the pixel value to the foreground components as shownin FIG. 90, based upon the processing increment and the virtual dividingnumber of the pixel value in the time direction.

[0841] For example, in the event that the number of pixel correspondingto the processing increment is 12, and the movement amount v in theshutter period is 5, the modeling unit 802 sets the virtual dividingnumber to 5, and selects a model made up of eight foreground componentsoverall, wherein the left-most positioned pixel contains one foregroundcomponent, the second pixel from the left contains two foregroundcomponents, the third pixel from the left contains three foregroundcomponents, the fourth pixel from the left contains four foregroundcomponents, the fifth pixel from the left contains five foregroundcomponents, the sixth pixel from the left contains five foregroundcomponents, the seventh pixel from the left contains five foregroundcomponents, the eighth pixel from the left contains five foregroundcomponents, the ninth pixel from the left contains four foregroundcomponents, the tenth pixel from the left contains three foregroundcomponents, the eleventh pixel from the left contains two foregroundcomponents, and the twelfth pixel from the left contains one foregroundcomponent.

[0842] Note that an arrangement may be made wherein the modeling unit802 does not select a model from the models stored beforehand, butrather generates a model based upon the movement vector and theprocessing increment in the event that the movement vector and theprocessing increment are supplied.

[0843] The modeling unit 802 supplies the selected model to anexpression generating unit 803.

[0844] The expression generating unit 803 generates a expression basedupon the model supplied from the modeling unit 802. The expressiongenerated by the expression generating unit 803 will be described in acase wherein the number of the foreground components is 8, the number ofpixel corresponding to the processing increment is 12, the movementamount v is 5, and the virtual dividing number is 5, with reference tothe model for foreground component image shown in FIG. 90.

[0845] In the event that the foreground component corresponding to theshutter period/v contained in the foreground component image are F01/vthrough F08/v, the relationships between F01/v through F08/v and thepixel values C01 through C12 are represented in Expression (78) throughExpression (89).

C01=F01/v  (78)

C02=F02/v+F01/v  (79)

C03=F03/v+F02/v+F01/v  (80)

C04=F04/v+F03/v+F02/v+F01/v  (81)

C05=F05/v+F04/v+F03/v+F02/v+F01/v  (82)

C06=F06/v+F05/v+F04/v+F03/v+F02/v  (83)

C07=F07/v+F06/v+F05/v+F04/v+F03/v  (84)

C08=F08/v+F07/v+F06/v+F05/v+F04/v  (85)

C09=F08/v+F07/v+F06/v+F05/v  (86)

C10=F08/v+F07/v+F06/v  (87)

C11=F08/v+F07/v  (88)

C12=F08/v  (89)

[0846] The expression generating unit 803 generates expressions bytransforming the generated expressions. The expressions generated by theexpression generating unit 803 are represented in Expression (90)through Expression (101).

C01=1·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (90)

C02=1·F01/v+1·F02/v+0·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (91)

C03=1·F01/v+1·F02/v+1·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (92)

C04=1·F01/v+1·F02/v+1·F03/v+1·F04/v+0·F05/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (93)

C05=1·F01/v+1·F02/v+1·F03/v+1·F04/v+1·F05/v+0·F06/v+0·F07/v+0·F08/v  (94)

C06=0·F01/v+1·F02/v+1·F03/v+1·F04/v+1·F05/v+1·F06/v+0·F07/v+0·F08/v  (95)

C07=0·F01/v+0·F02/v+1·F03/v+1·F04/v+1·F05/v+1·F06/v+1·F07/v+0·F08/v  (96)

C08=0·F01/v+0·F02/v+0·F03/v+1·F04/v+1·F05/v+1·F06/v+1·F07/v+1·F08/v  (97)

C09=0·F01/v+0·F02/v+0·F03/v+0·F04/v+1·F05/v+1·F06/v+1·F07/v+1·F08/v  (98)

C10=0·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+1·F06/v+1·F07/v+1·F08/v  (99)

C11=0·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+0·F06/v+1·F07/v+1·F08/v  (100)

C12=0·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+1·F08/v  (101)

[0847] Expression (90) through Expression (101) may be represented aswith Expression (102). $\begin{matrix}{{C\quad j} = {\sum\limits_{i = 01}^{08}{a\quad i\quad {j \cdot F_{i/v}}}}} & (102)\end{matrix}$

[0848] In Expression (102), j denotes the pixel position. In thisexample, j has one of the values between 1 and 12. Also, i denotes theposition of the foreground value. In this example, i has one of thevalues between 1 and 8. Corresponding to the values of i and j, aij hasone of the values of 0 or 1.

[0849] Taking margin of error into consideration, Expression (102) maybe represented as with Expression (103). $\begin{matrix}{{C\quad j} = {{\sum\limits_{i = 01}^{08}{a\quad i\quad {j \cdot F_{i/v}}}} + {e\quad j}}} & (103)\end{matrix}$

[0850] In Expression (103), ej denotes the margin of error contained inthe pixel of interest, Cj.

[0851] Expression (103) can be rewritten into Expression (104)$\begin{matrix}{{e\quad j} = {{C\quad j} - {\sum\limits_{i = 01}^{08}{a\quad i\quad {j \cdot F_{i/v}}}}}} & (104)\end{matrix}$

[0852] Note that in order to use the least square method, thesquared-sum E of the margin of error is defined as represented inExpression (105). $\begin{matrix}{E = {\sum\limits_{j = 01}^{12}{e\quad j^{2}}}} & (105)\end{matrix}$

[0853] To minimize margin of error, the value of the partial derivativefrom the variable Fk as to the squared-sum of the margin of error Eshould become 0. Fk is obtained so as to satisfy Expression (106).$\begin{matrix}\begin{matrix}{\frac{\partial E}{{\partial F}\quad k} = {2 \cdot {\sum\limits_{j = 01}^{12}{e\quad {j \cdot \frac{{\partial e}\quad j}{{\partial F}\quad k}}}}}} \\{= {2 \cdot {\sum\limits_{j = 01}^{12}\left\{ {{\left( {{C\quad j} - {\sum\limits_{i = 01}^{08}{a\quad i\quad {j \cdot F_{i/v}}}}} \right) \cdot \left( {- a_{k\quad {j/v}}} \right)} = 0} \right.}}}\end{matrix} & (106)\end{matrix}$

[0854] In Expression (106), the movement amount v is a fixed value, soExpression (107) can be derived. $\begin{matrix}{{\sum\limits_{j = 01}^{12}{a_{k\quad j}\quad \cdot \left( {{C\quad j} - {\sum\limits_{i = 01}^{08}{a\quad i\quad {j \cdot F_{i/v}}}}} \right)}} = 0} & (107)\end{matrix}$

[0855] Developing Expression (107) and transposing arguments, Expression(108) is obtained. $\begin{matrix}{{\sum\limits_{j = 01}^{12}\left( {a_{k\quad j} \cdot {\sum\limits_{i = 01}^{08}{a\quad i\quad {j \cdot F}\quad i}}} \right)} = {v \cdot {\sum\limits_{j = 01}^{12}{{a_{k\quad j} \cdot C}\quad j}}}} & (108)\end{matrix}$

[0856] Expression (108) is developed into eight expressions, each ofwhich is obtained by substituting one of the integers between 1 and 8for k in Expression (108). The obtained eight expressions may berepresented in one expression by a matrix. The expression is referred toas a normal equation.

[0857] An example of the normal equation generated by the expressiongenerating unit 803 based upon such a least square method is representedin Expression (109). $\begin{matrix}{{\begin{bmatrix}5 & 4 & 3 & 2 & 1 & 0 & 0 & 0 \\4 & 5 & 4 & 3 & 2 & 1 & 0 & 0 \\3 & 4 & 5 & 4 & 3 & 2 & 1 & 0 \\2 & 3 & 4 & 5 & 4 & 3 & 2 & 1 \\1 & 2 & 3 & 4 & 5 & 4 & 3 & 2 \\0 & 1 & 2 & 3 & 4 & 5 & 4 & 3 \\0 & 0 & 1 & 2 & 3 & 4 & 5 & 4 \\0 & 0 & 0 & 1 & 2 & 3 & 4 & 5\end{bmatrix}\begin{bmatrix}{F01} \\{F02} \\{F03} \\{F04} \\{F05} \\{F06} \\{F07} \\{F08}\end{bmatrix}} = {v \cdot \begin{bmatrix}{\sum\limits_{i = 08}^{12}{C\quad i}} \\{\sum\limits_{i = 07}^{11}{C\quad i}} \\{\sum\limits_{i = 06}^{10}{C\quad i}} \\{\sum\limits_{i = 05}^{09}{C\quad i}} \\{\sum\limits_{i = 04}^{08}{C\quad i}} \\{\sum\limits_{i = 03}^{07}{C\quad i}} \\{\sum\limits_{i = 02}^{06}{C\quad i}} \\{\sum\limits_{i = 01}^{05}{C\quad i}}\end{bmatrix}}} & (109)\end{matrix}$

[0858] In the event that Expression (109) is represented by A F=v·C,then C, A, and v are known, and F is unknown. Also, while A and v areknown at the point of modeling, C becomes known by inputting the pixelvalue in addition operation.

[0859] The margin of error contained in the pixel C is dispersed bycalculating the foreground components by the normal equation based uponthe least square method.

[0860] The expression generating unit 803 supplies the normal equationgenerated as described above, to the addition unit 804.

[0861] The addition unit 804 sets the pixel value C contained in theforeground component image for the expression of the matrix suppliedfrom the expression generating unit 803 based upon the processingincrement supplied from the processing increment decision unit 801. Theaddition unit 804 supplies the matrix which the pixel value C is setfor, to the computing unit 805.

[0862] The computing unit 805 calculates the foreground component Fi/vwhich has been subjected to removal of the movement blurring by theprocessing based upon the method such as the sweeping method(Gauss-Jordan elimination), calculates Fi corresponding to one of theintegers i between 0 and 8, which is the pixel value of the foregroundwhich has been subjected to removal of the movement blurring, andoutputs the foreground component image which has been subjected toremoval of the movement blurring, which is made up of Fi which is thepixel value which has been subjected to removal of the movement blurringas shown by way of an example, shown in FIG. 91.

[0863] Note that in the foreground component image which has beensubjected to removal of the movement blurring shown in FIG. 91, each ofC03 through C10 is set to each of F01 through F08 so as not to changethe position of the foreground component image with regard to thescreen, which can correspond to an arbitrary position.

[0864] Also, as shown in FIG. 92, for example, in the event that thenumber of pixel corresponding to the processing increment is 8 and themovement amount v is 4, the movement blurring removal unit 106 generatesa matrix expression represented in Expression (110). $\begin{matrix}{{\begin{bmatrix}4 & 3 & 2 & 1 & 0 \\3 & 4 & 3 & 2 & 1 \\2 & 3 & 4 & 3 & 2 \\1 & 2 & 3 & 4 & 3 \\0 & 1 & 2 & 3 & 4\end{bmatrix}\begin{bmatrix}{F01} \\{F02} \\{F03} \\{F04} \\{F05}\end{bmatrix}} = {v \cdot \begin{bmatrix}{\sum\limits_{i = 05}^{08}{C\quad i}} \\{\sum\limits_{i = 04}^{07}{C\quad i}} \\{\sum\limits_{i = 03}^{06}{C\quad i}} \\{\sum\limits_{i = 02}^{05}{C\quad i}} \\{\sum\limits_{i = 01}^{04}{C\quad i}}\end{bmatrix}}} & (110)\end{matrix}$

[0865] The movement blurring removal unit 106 calculates Fi which is thepixel value which has been subjected to adjustment of movement blurringby forming expressions of which number corresponds to the length of theprocessing increment. In the same way, in the event that the number ofpixel contained in the processing increment is one hundred, Fi iscalculated by generating expressions corresponding to the one hundredpixels.

[0866] As described above, the movement blurring removal unit 106generates expressions corresponding to the movement amount v and theprocessing increment, sets pixel values of the foreground componentimage for the generated expressions, and calculates an foregroundcomponent image which has been subjected to removal of movementblurring.

[0867] The processing for removal of movement blurring contained in theforeground component image by the movement blurring removal unit 106will now be descried with reference to the flowchart shown in FIG. 93.

[0868] In Step S801, the processing increment decision unit 801 of themovement blurring removal unit 106 generates the processing incrementbased upon the movement vector and the region information, and suppliesthe generated processing increment to the modeling unit 802.

[0869] In Step S802, the modeling unit 802 of the movement blurringremoval unit 106 performs selecting or generating of the modelcorresponding to the movement amount v and the processing increment. InStep S803, the expression generating unit 803 creates the normalequation based upon the selected model.

[0870] In Step S804, the addition unit 804 sets the pixel values of theforeground component image for the created normal equation. In StepS805, the addition unit 804 judges whether or not the pixel values ofall the pixels corresponding to the processing increment are set, and inthe event that judgment is made that not all the pixel values of thepixels corresponding to the processing increment have been set, the flowreturns to Step S804 and repeats the processing of setting the pixelvalues for the normal equation.

[0871] In the event that judgment is made that all the pixel values ofthe pixels of the processing increment have been set in Step S805, theflow proceeds to Step S806, the computing unit 805 calculates the pixelvalues of the foreground which has been subjected to removal of movementblurring based upon the normal equation wherein the pixel valuessupplied from the addition unit 804 are set, and the processing ends.

[0872] As described above, the movement blurring removal unit 106 canremove movement blurring from the foreground image containing themovement blurring based upon the movement vector and the regioninformation.

[0873] That is to say, movement blurring contained in the pixel valueswhich are the sampled data, can be removed.

[0874] The correction of the background component image by thecorrection unit 107 will now be described.

[0875]FIG. 94 is a diagram which illustrates an example of the model ofthe background component image corresponding to the model of theforeground component image shown by way of an example shown in FIG. 90.

[0876] As shown in FIG. 94, pixel values of the pixels of the backgroundcomponent image corresponding to the mixed region in the original inputimage have been subjected to removal of the foreground components, andaccordingly the pixel values are made up of a small number of backgroundcomponents as compared with the pixels corresponding to the backgroundregion in the original input image, corresponding to the mixture ratioα.

[0877] For example, in the background component image shown by way of anexample shown in FIG. 94, the pixel value C01 is made up of fourbackground components B02/Vs, the pixel value C02 is made up of threebackground components B03/Vs, the pixel value C03 is made up of twobackground components B04/Vs, and the pixel value C04 is made up of onebackground component B05/V.

[0878] Also, with the background component image shown by way of anexample shown in FIG. 94, the pixel value C09 is made up of onebackground component B10/V, thee pixel value C10 is made up of twobackground components B11/Vs, the pixel value C11 is made up of threebackground components B12/Vs, and the pixel value C12 is made up of fourbackground components B13/Vs.

[0879] As described above, the pixel value of a pixel corresponding tothe mixed region in the original input image is made up of a smallnumber of background components as compared with the pixel correspondingto the background region in the original input image, and accordinglythe image corresponding to the mixed region in the foreground componentimage becomes a dark image, for example, as compared with the image ofthe background region.

[0880] The correction unit 107 corrects pixel values of the pixelscorresponding to the mixed region in the background component image bymultiplying each of pixel values of the pixels corresponding to themixed region in the background component image by a constantcorresponding to the mixture ratio α.

[0881] For example, in the event that the background component imageshown in FIG. 94 is input, the correction unit 107 multiplies the pixelvalue C01 by 5/4, multiplies the pixel value C02 by 5/3, multiplies thepixel value C11 by 5/3, and multiplies the pixel value C12 by 5/4. Inorder to match the pixel position of the foreground component imagewhich has been subjected to removal of movement blurring shown by way ofan example shown in FIG. 91, the correction unit 107 sets the pixelvalue C03 through pixel value C11 to 0.

[0882] The correction unit 107 outputs a background component imagewhich has been subjected to correction of the pixel values of pixelscorresponding to the mixed region shown by way of an example shown inFIG. 95.

[0883] As described above, the correction unit 107 corrects pixel valuesof the pixels corresponding to the mixed region in the backgroundcomponent image, and also adjusts the pixel position with regard to theforeground component image which has been subjected to removal ofmovement blurring.

[0884]FIG. 96 is a block diagram which illustrates the configuration ofthe movement-blurring-removed-image processing unit 108 for generating acoefficient set which is used in class classification adaptationprocessing for generating an even higher resolution image in the spatialdirection. For example, the movement-blurring-removed-image processingunit 108 of which the configuration is shown in FIG. 96 generates acoefficient set which is used in class classification adaptationprocessing for generating a HD image from a SD image based upon theinput HD image.

[0885] Background component tutor image frame memory 1001 stores thecorrected background component image of the tutor image supplied fromthe correction unit 107. The background component tutor image framememory 1001 supplies the stored background component image of the tutorimage to a weighted averaging unit 1003-1 and a learning unit 1006-1.

[0886] Foreground component tutor image frame memory 1002 stores theforeground component image which has been subjected to removal ofmovement blurring of the tutor image supplied from the movement blurringremoval unit 106. The foreground component tutor image frame memory 1002supplies the stored foreground component image of the tutor image to aweighted averaging unit 1003-2 and a learning unit 1006-2.

[0887] The weighted averaging unit 1003-1 generates a SD image which isa student image by one-quarter weighted-averaging the backgroundcomponent image of a tutor image which is a HD image, and supplies thegenerated SD image to background component student image frame memory1004.

[0888] For example, the weighted averaging unit 1003-1 takes four pixelsof 2×2 (width x height) (which are portions represented by white circlesin the drawing) as one increment in the tutor image as shown in FIG. 97,adds pixel values of four pixel of each increment, and the sum isdivided by 4. The weighted averaging unit 1003-1 sets the one-quarterweighted averaged results described above for the pixel of the studentimage positioned at the center of each increment (which are the portionsrepresented by solid circles in the drawing).

[0889] The background component student image frame memory 1004 storesthe student image corresponding to the background component image of thetutor image supplied from the weighted averaging unit 1003-1. Thebackground component student image frame memory 1004 supplies thestudent image corresponding to the background component image of thetutor image stored therein to the learning unit 1006-1.

[0890] The weighted averaging unit 1003-2 generates a SD image which isa student image by one-quarter weighted-averaging the foregroundcomponent image of a tutor image which is a HD image supplied from theforeground component tutor image frame memory 1002, for example, andsupplies the generated SD image to foreground component student imageframe memory 1005.

[0891] The foreground component student image frame memory 1005 storesthe student image which is a SD image, corresponding to the foregroundcomponent image of the tutor image supplied from the weighted averagingunit 1003-2. The foreground component student image frame memory 1005supplies the student image corresponding to the foreground componentimage of the tutor image stored therein to the learning unit 1006-2.

[0892] The learning unit 1006-1 generates coefficient sets correspondingto the background component image based upon the background componentimage of the tutor image supplied from the background component tutorimage frame memory 1001 and the student image corresponding to thebackground component image of the tutor image supplied from thebackground component student image frame memory 1004, and supplies thegenerated coefficient sets to coefficient set memory 1007.

[0893] The learning unit 1006-2 generates coefficient sets correspondingto the foreground component image based upon the foreground componentimage of the tutor image supplied from the foreground component tutorimage frame memory 1002 and the student image corresponding to theforeground component image of the tutor image supplied from theforeground component student image frame memory 1005, and supplies thegenerated coefficient sets to coefficient set memory 1007.

[0894] The coefficient set memory 1007 stores the coefficient setscorresponding to the background component image supplied from thelearning unit 1006-1 and the foreground component image supplied fromthe learning unit 1006-2.

[0895] In the event that there is no need to differentiate the learningunit 1006-1 and the learning unit 1006-2, individually, these will besimply referred to as a learning unit 1006 below.

[0896]FIG. 98 is a block diagram which illustrates the configuration ofthe learning unit 1006.

[0897] A class classification unit 1031 comprises a class tap obtainingunit 1051 and a waveform classification unit 1052, and classifies thepixel of interest of the input image. The class tap obtaining unit 1051obtains a predetermined number of class taps which are pixel values ofthe student image corresponding to the pixel of interest, and suppliesthe obtained class taps to the waveform classification unit 1052.

[0898] For example, in FIG. 97, in the event that the pixel which is thei'th from the top and the j'th from the left in the student image (whichis a portion indicated by a solid circle in the drawing) is representedby X_(ij), the class tap obtaining unit 1051 obtains a class tap whichconsists of nine pixels in total, i.e., the eight pixels at left-top,right-top, left, right, bottom-left, bottom, and right-bottom, adjacentto the pixel of interest X_(ij), X_((i−1)(j−1)), X_((i−1)j),X_((i−1)(j+1)), X_(i(j−1), X_(i(j+1)), X_((i−1)(j−1)), X_((i−1)j),X_((i+1)(j+1)), and also the pixel of interest. The class tap issupplied to the waveform classification unit 1052.

[0899] Note that in this case, while the class tap consists of asquare-shaped block made up of 3×3 pixels, this needs not be a square;rather other arbitrary shapes may be used, for example, arectangle-shape, a cross-shape, or the like. Also, the number of pixelsmaking up the class tap is not restricted to nine pixels of 3×3 pixels.

[0900] The waveform classification unit 1052 performs classclassification processing wherein the input signals are classified intoseveral classes based upon the features thereof, and classifies thepixel of interest into one class based upon the class taps. For example,the waveform classification unit 1052 classifies the pixel of interestinto one of 512 classes, and supplies the class No. corresponding to theclassified class to a prediction tap obtaining unit 1032.

[0901] Here, the class classification processing will now be describedbriefly.

[0902] Now, let us say that a given pixel of interest and three adjacentpixels make up a class tap which consists of 2×2 pixels as shown in FIG.99A, and each pixel is represented by 1 bit (has a level of either 0 or1). The solid circle shown in FIG. 99A denotes the pixel of interest. Inthis case, four pixel block of 2×2 pixels containing the pixel ofinterest can be classified into 16 (=(2¹)⁴) patterns by the leveldistribution for each pixel as shown in FIG. 99B. In FIG. 99B, whitecircles denote 0, and solid circles denote 1. Accordingly, in this case,the pixel of interest can be classified into sixteen patterns, whereinpattern-classification is the class-classification processing, and theprocessing is performed by the class classification unit 1031.

[0903] Note that an arrangement may be made wherein the classclassification processing is performed by taking the activity(complexity of the image) (intensity of change) of the image (class tap)into consideration.

[0904] Here, each pixel is generally appropriated around 8 bits. Also,with the present embodiment, the class tap consists of nine pixels of3×3 pixels as described above. Accordingly, performing classclassification processing for such a class tap as an object, the classtap would result the class tap being classified into a great number ofclasses of which number is (2⁸)⁹.

[0905] Accordingly, with the present embodiment, the ADCR processing isperformed for the class tap by the waveform classification unit 1052,and this reduces the number of classes by reducing the number of bits ofthe pixels making up the class tap.

[0906] In order to simplify description, with a class tap which consistsof four pixels arrayed in a line as shown in FIG. 110A, the maximumvalue of the pixel value MAX and the minimum value of the pixel valueMIN are detected in the ADRC processing. DR=MAX−MIN is then taken as thelocal dynamic range in the block which consists of a class tap, and thepixel values of the pixels making up the block of the class tap isre-quantized into K bits based upon the dynamic range DR.

[0907] That is to say, the minimum value MIN is subtracted from eachpixel value within the block, and the subtraction value is divided byDR/2^(k). The division value obtained as a result is converted into thecode (ADRC code) corresponding thereto. Specifically, for example, inthe event of taking K as 2, judgment is made which of ranges obtained bydividing the dynamic range DR into four (=2²) equal parts the divisionvalue belongs to, as shown in FIG. 100B, and upon the division valuebelonging to the range of the bottom-most level, the range of the secondlevel from the bottom, the range of the third level from the bottom, orthe range of upper-most level, the division value is encoded into 2-bitcode such as 00B, 01B, 10B, or 11B (B indicates a binary number),respectively, for example. Decoding is then performed on the decodingside by the ADRC code 00B, 01B, 10B, or 11B being converted into themedian in the range of the most-bottom level L₀₀, the median in therange of the second level from the bottom L₀₁, the median in the rangeof the third level from the bottom L₁₀, or the median in the range ofthe most-upper level L₁₁, wherein the ranges are obtained by dividingthe dynamic range DR into four equal parts, and the minimum value MINbeing added to the converted value.

[0908] Here, the ADRC processing described above is referred to asnon-edge-matching.

[0909] Note that details with regard to the ADRC processing aredisclosed in Japanese Unexamined Patent Application Publication No.3-53778, which has been applied by the present applicant, and so forth,for example.

[0910] The number of classes can be reduced by performing the ADRCprocessing which performs re-quantizing with the number of bits lessthan the number of bits appropriated to pixels making up the class tapas described above, and the ADRC processing described above is performedby the waveform classifying unit 1052.

[0911] While the class classification processing is performed based uponthe ADRC code by the waveform classification unit 1052 in the presentembodiment, an arrangement may be made wherein the class classificationprocessing is performed with regard to the data which has been subjectedto DPCM (Predictive Coding), BTC (Block Truncation Coding), VQ (VectorQuantizing), DCT (Disperse Cosine Transformation), Hadamardtransformation, or the like.

[0912] The prediction tap obtaining unit 1032 obtains the prediction tapwhich is the increment for calculation of the predicted value of theoriginal image (tutor image) corresponding to the class based upon theclass No. from pixels of the student image, and supplies the obtainedprediction tap and the class No. to a corresponding pixel obtaining unit1033.

[0913] For example, in FIG. 97, let us say that pixel values of ninepixels of 2×2 centered on the pixel X_(ij) in the student image (whichis denoted by a solid circle in the drawing) in the original image(tutor image) are represented as Y_(ij)(1), Y_(ij)(2), Y_(ij)(3),Y_(ij)(4), respectively, in the direction from the far left to theright, and in the direction from the top to the bottom, the predictiontap obtaining unit 1032 obtains a square-shaped prediction tap whichconsists of nine pixels of 3×3, X_((i−1)(j−1)), X_((i−1)j),X_((i−1)(j+1)) X_(i(j−1)), X_(ij), X_(i(j+1)), X_((i+1)(j−1)),X_((i+1)j), X_((i+1)(j+1)), centered on the pixel X_(ij) in the studentimage, for example, for calculating the coefficients which are necessaryfor calculation of the predicted values of the pixels Y_(ij)(1) throughY_(ij)(4).

[0914] Specifically, for example, the pixel X₂₂, X₂₃, X₂₄, X₃₂, X₃₃,X₃₄, X₄₂, X₄₃, X₄₄ make up the prediction tap for calculating thecoefficients which are necessary for calculation of the predicted valuesof four pixels of Y₃₃(1) through Y₃₃(4) in the tutor image, which areenclosed by a quadrangle in FIG. 97, (in this case, the pixel ofinterest is X₃₃).

[0915] The corresponding pixel obtaining unit 1033 obtains pixel valuesof the pixels in the tutor image corresponding to the pixel values whichare to be predicted based upon the prediction tap and the class No., andsupplies the prediction tap, the class No., and the obtained pixelvalues of the pixels in the tutor image corresponding to the pixelvalues which are to be predicted to a normal equation generating unit1034.

[0916] For example, in the event that the coefficients necessary forcalculation of the predicted values of four pixels of Y₃₃(1) throughY₃₃(4) in the tutor image, the corresponding pixel obtaining unit 1033obtains the pixel values of the pixels, Y₃₃(1) through Y₃₃(4) as thepixels in the tutor image corresponding to the pixel values which are tobe predicted.

[0917] The normal equation generating unit 1034 generates normalequations for calculating a coefficient set which is used in theadaptation processing, corresponding to the correlation between theprediction tap and the pixels which are to be predicted, based upon theprediction tap, the class No., and the obtained pixel values which areto be predicted, and supplies the generated normal equations to acoefficient calculation unit 1035 along with the class No.

[0918] The coefficient calculation unit 1035 calculates a coefficientset which is used in the adaptation processing, corresponding to theclassified class, by solving the normal equations supplied from thenormal equation generating unit 1034. The coefficient calculation unit1035 supplies the calculated coefficient set to the coefficient setmemory 1007 along with the class No.

[0919] An arrangement may be made wherein the normal equation generatingunit 1034 generates a matrix corresponding to such normal equations, andthe coefficient calculation unit 1035 calculates a coefficient set basedupon the generated matrix.

[0920] Here, the adaptation processing will be described.

[0921] For example, let us now consider obtaining a predicted value E[y]of the pixel value y in the tutor image from a linear one-dimensionalcombination model defined by linear combination of pixel values ofseveral nearby pixels x₁, x₂, . . . (which will be referred to asstudent data as appropriate) and predetermined prediction coefficientsw₁, w₂, . . . . In this case, the predicted value E[y] may berepresented in the following Expression.

E[y]=w ₁ x ₁ +w ₂ x ₂+ . . .   (111)

[0922] Accordingly, upon defining the matrix W which consists of a setof the prediction coefficients, the matrix X which consists of a set ofthe student data, and the matrix Y′ which consists of a set of thepredicted values E[y] as $\begin{matrix}{X = \begin{pmatrix}x_{11} & x_{12} & \cdots & x_{1n} \\x_{21} & x_{22} & \cdots & x_{2n} \\\cdots & \cdots & \cdots & \cdots \\x_{m1} & x_{m2} & \cdots & x_{m\quad n}\end{pmatrix}} \\{{{{{W = \begin{pmatrix}\begin{matrix}\begin{matrix}w_{1} \\w_{2}\end{matrix} \\\cdots\end{matrix} \\w_{n}\end{pmatrix}},Y}’} = \begin{pmatrix}\begin{matrix}\begin{matrix}{E\left\lbrack y_{1} \right\rbrack} \\{E\left\lbrack y_{2} \right\rbrack}\end{matrix} \\\cdots\end{matrix} \\{E\left\lbrack y_{m} \right\rbrack}\end{pmatrix}},}\end{matrix}$

[0923] the following observation expression holds.

XW=Y′  (112)

[0924] Let us now consider obtaining the predicted value E[y] near thepixel value y of the original image by applying the least square methodto the observation expression. In this case upon defining the matrix Ywhich consists of a set of pixel values y in the original image (whichwill be referred to as tutor data as appropriate) and the matrix E whichconsists of a set of the residuals e of the predicted values E[y] withregard to the pixel values y in the original image as${E = \begin{pmatrix}e_{1} \\e_{2} \\\cdots \\e_{m}\end{pmatrix}},{Y = \begin{pmatrix}y_{1} \\y_{2} \\\cdots \\y_{m}\end{pmatrix}},$

[0925] the following residual expression holds from Expression (112).

XW=Y+E  (113)

[0926] In this case, the prediction coefficients w_(i) for obtaining thepredicted value E[y] near the pixel value y in the original image can beobtained by minimizing the squared margin of error$\sum\limits_{i = 1}^{m}{e_{i}^{2}.}$

[0927] Accordingly, in a case that the derivative of the above-describedsquared margin of error from the prediction coefficient w_(i) is 0, thatis to say, in a case that the prediction coefficient w_(i) satisfies thefollowing expression, the prediction coefficient w_(i) is the optimalvalue for obtaining the predicted values E[y] near the pixel value y inthe original image. $\begin{matrix}{{{e_{1}\frac{\partial e_{1}}{\partial w_{i}}} + {e_{2}\frac{\partial e_{2}}{\partial w_{i}}} + \ldots + {e_{m}\frac{\partial e_{m}}{\partial w_{i}}}} = {0\quad \left( {{i = 1},2,\quad \ldots \quad,n} \right)}} & (114)\end{matrix}$

[0928] Here, the following expression holds by differentiatingExpression (113) by the prediction coefficient w_(i). $\begin{matrix}{{\frac{\partial e_{i}}{\partial w_{1}} = x_{i\quad 1}},{\frac{\partial e_{i}}{\partial w_{2}} = x_{i\quad 2}},\quad {{\ldots \quad \frac{\partial e_{i}}{\partial w_{n}}} = x_{i\quad n}},\left( {{i = 1},2,\quad \ldots \quad,m} \right)} & (115)\end{matrix}$

[0929] Expression (116) is obtained from Expression (114) and Expression(115). $\begin{matrix}{{{\sum\limits_{i = 1}^{m}\quad {e_{i}x_{i1}}} = 0},{{\sum\limits_{i = 1}^{m}\quad {e_{i}x_{i2}}} = 0},\quad {{\ldots \quad {\sum\limits_{i = 1}^{m}\quad {e_{i}x_{i\quad n}}}} = 0}} & (116)\end{matrix}$

[0930] Moreover, taking the relationship between the student data x, theprediction coefficient w, the tutor data y, and the residuals e in theresidual expression (113), into consideration, the following normalequations can be obtained from Expression (116). $\begin{matrix}\left\{ \begin{matrix}{{{\left( {\sum\limits_{i = 1}^{m}\quad {x_{i1}x_{i1}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{m}\quad {x_{i1}x_{i2}}} \right)w_{2}} + \ldots + {\left( {\sum\limits_{i = 1}^{m}\quad {x_{i1}x_{i\quad n}}} \right)w_{n}}} = \left( {\sum\limits_{i = 1}^{m}\quad {x_{i1}y_{i}}} \right)} \\{{{\left( {\sum\limits_{i = 1}^{m}\quad {x_{i2}x_{i1}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{m}\quad {x_{i2}x_{i2}}} \right)w_{2}} + \ldots + {\left( {\sum\limits_{i = 1}^{m}\quad {x_{i2}x_{i\quad n}}} \right)w_{n}}} = \left( {\sum\limits_{i = 1}^{m}\quad {x_{i2}y_{i}}} \right)} \\\cdots \\{{{\left( {\sum\limits_{i = 1}^{m}\quad {x_{i\quad n}x_{i1}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{m}\quad {x_{i\quad n}x_{i2}}} \right)w_{2}} + \ldots + {\left( {\sum\limits_{i = 1}^{m}\quad {x_{i\quad n}x_{i\quad n}}} \right)w_{n}}} = \left( {\sum\limits_{i = 1}^{m}\quad {x_{i\quad n}y_{i}}} \right)}\end{matrix} \right. & (117)\end{matrix}$

[0931] As many normal equations represented in Expression (117) can beformed as the number of the prediction coefficients w which are to beobtained, and accordingly the optimal prediction coefficients w can beobtained by solving Expression (117). Note that Expressions (117) can besolved by applying the sweeping method (Gauss-Jordan elimination), forexample.

[0932] The adaptation processing consists of the optimal predictioncoefficients w being obtained for each class, and the predicted valuesE[y] near the pixel values y in the tutor image being obtained byExpression (111) using the prediction coefficients w.

[0933] The normal equation generating unit 1034 generates the normalequations for calculating the most suitable prediction coefficients wfor each class, and the coefficient calculation unit 1035 calculates theprediction coefficients w based upon the generated normal equations.

[0934] Note that the adaptation processing is different from theinterpolation processing with regard to the components which are notcontained in the thinned out image and are contained in the originalimage being reproduced. That is to say, while in the event of takingonly Expression (111) into consideration, the adaptation processing isthe same as the interpolation processing using the interpolation filter,the prediction coefficients w corresponding to the tap coefficients ofthe interpolation filter is obtained by learning as if it were, usingthe tutor data y, and accordingly the adaptation processing canreproduce the components contained in the original image. Accordingly,it can be said that the adaptation processing acts to create an image,as if it were.

[0935]FIG. 101 is a diagram which describes a coefficient set generatedby the movement-blurring-removed-image processing unit 108 of whichconfiguration is shown in FIG. 96. The region specifying unit 103specifies the foreground region, the background region, the coveredbackground region, and the uncovered background region in the inputimage.

[0936] The input image wherein the regions have been specified and themixture ratio α has been detected by the mixture ratio calculation unit104, is separated into the foreground component image and the backgroundcomponent image by the foreground/background separation unit 105.

[0937] The movement blurring is removed from the separated foregroundcomponent image by the movement blurring removal unit 106. The pixelvalues corresponding to the mixed region in the separated backgroundcomponent image are corrected by the correction unit 107 correspondingto the removal of the movement blurring of the foreground componentimage.

[0938] The movement-blurring-removed-image processing unit 108calculates a coefficient set corresponding to the foreground componentimage and a coefficient set corresponding to the background componentimage, respectively, based upon the foreground component image which hasbeen subjected to removal of movement blurring and the backgroundcomponent image which has been subjected to correction.

[0939] That is to say, the learning unit 1006-1 calculates a coefficientset corresponding to the background component image based upon theseparated and corrected background component image, and the learningunit 1006-2 calculates a coefficient set corresponding to the foregroundcomponent image based upon the foreground component image which has beensubjected to separation and removal of movement blurring.

[0940] The coefficient set corresponding to the background componentimage is used for predicting the pixel values of the image correspondingto the background component image in the class classification adaptationprocessing for predicting the pixel values, which is to be applied tothe separated and corrected background component image.

[0941] The coefficient set corresponding to the foreground componentimage is used for predicting the pixel values of the image correspondingto the foreground component image in the class classification adaptationprocessing for predicting the pixel values, which is to be applied tothe foreground component image which has been subjected to separationfrom the input image and removal of movement blurring.

[0942] The movement blurring is added to the predicted imagecorresponding to the foreground component image. The predicted imagecorresponding to the background component image is correctedcorresponding to addition of the movement blurring to the foregroundcomponent image.

[0943] The predicted image corresponding to the corrected backgroundcomponent image and the predicted image corresponding to the foregroundcomponent image which has been subjected to addition of the movementblurring, are synthesized into a single predicted image.

[0944] Referring to the flowchart shown in FIG. 102, description will bemade with regard to the processing of learning for generating acoefficient set which is used in prediction of the pixel values by theclass classification adaptation processing in themovement-blurring-removed-image processing unit 108 of whichconfiguration is shown in FIG. 96.

[0945] In Step S1001, the weighted averaging unit 1003-1 and theweighted averaging unit 1003-2 generate a student image corresponding tothe background component image and a student image corresponding to theforeground component image. That is to say, the weighted averaging unit1003-1 generates a student image corresponding to the backgroundcomponent image of the tutor image by one-quarter weighted-averaging ofthe background component image of the tutor image stored in thebackground component tutor image frame memory 1001, for example.

[0946] The weighted averaging unit 1003-2 generates a student imagecorresponding to the foreground component image of the tutor image byone-quarter weighted-averaging of the foreground component image of thetutor image stored in the foreground component tutor image frame memory1002, for example.

[0947] In Step S1002, the learning unit 1006-1 generates a coefficientset corresponding to the background component image based upon thebackground component image of the tutor image stored in the backgroundcomponent tutor image frame memory 1001 and the student imagecorresponding to the background component image of the tutor imagestored in the background component student image frame memory 1004.Details of the processing for generating of a coefficient set in StepS1002 will be described later with reference to the flowchart shown inFIG. 103.

[0948] In Step S1003, the learning unit 1006-2 generates a coefficientset corresponding to the foreground component image based upon theforeground component image of the tutor image stored in the foregroundcomponent tutor image frame memory 1002 and the student imagecorresponding to the foreground component image of the tutor imagestored in the foreground component student image frame memory 1005.

[0949] In Step S1004, the learning unit 1006-1 and the learning unit1006-2 output a coefficient set corresponding to the backgroundcomponent image and a coefficient set corresponding to the foregroundcomponent image to the coefficient set memory 1007, respectively. Thecoefficient set memory 1007 stores the coefficient set corresponding tothe background component image, or the coefficient set corresponding tothe foreground component image, and then the processing ends.

[0950] As described above, the movement-blurring-removed-imageprocessing unit 108 of which configuration is shown in FIG. 96 cangenerate a coefficient set corresponding to the background componentimage and a coefficient set corresponding to the foreground componentimage.

[0951] Note that it is needless to say that the processing in Step S1002and Step S1003 may be performed serially or in parallel.

[0952] Referring to FIG. 103, the processing for generating of acoefficient set corresponding to the background component imageperformed by the learning unit 1006-1, corresponding to Step S1002, willnow be described.

[0953] In Step S1021, the learning unit 1006-1 judges whether or notthere are any unprocessed pixels in the student image corresponding tothe background component image, and in the event that judgment is madethat there are unprocessed pixels in the student image corresponding tothe background component image, the flow proceeds to Step S1022, and thepixel of interest is obtained from the student image corresponding tothe background component image in raster scan sequence.

[0954] In Step S1023, the class tap obtaining unit 1051 of the class tapclassification unit 1031 obtains a class tap corresponding to the pixelof interest from the student image stored in the background componentstudent image frame memory 1004. In Step S1024, the waveformclassification unit 1052 of the class classification unit 1031 appliesthe ADRC processing to the class tap, this reduces the number of bits ofpixels making up the class tap, and the pixel of interest is classified.In Step S1025, the prediction tap obtaining unit 1032 obtains aprediction tap corresponding to the pixel of interest from the studentimage stored in the background component student image frame memory 1004based upon the classified class.

[0955] In Step S1026, the corresponding pixel obtaining unit 1033obtains pixels corresponding to the pixel value which is to be predictedfrom the background component image of the tutor image stored in thebackground component tutor image frame memory 1001 based upon theclassified class.

[0956] In Step S1027, the normal equation generating unit 1034 adds thepixel values of pixels corresponding to the prediction tap and the pixelvalue which is to be predicted to the matrix for each class based uponthe classified class, the flow returns to Step S1021, and the learningunit 1006-1 repeats judgment whether or not unprocessed pixels exist.The prediction tap and the matrix for each class to which the pixelvalues of pixels corresponding to the prediction tap and the pixel valuewhich is to be predicted is added, corresponds to the normal equationsfor calculating a coefficient set for each class.

[0957] In Step S1021, in the event that judgment is made that there areno unprocessed pixels in the student image, the flow proceeds to StepS1028, and the normal equation generating unit 1034 supplies the matrixfor each class for which the pixel values of the pixel corresponding tothe prediction tap and the pixel value which is to be predicted is set,to the coefficient calculation unit 1035. The coefficient calculationunit 1035 calculates a coefficient set for each class corresponding tothe background component image by solving the matrix for each class, forthe pixel values of pixels corresponding to the prediction tap and thepixel value which is to be predicted are set.

[0958] Note that the coefficient set is not restricted to predicting thepixel values by linear prediction, rather, an arrangement may be madewherein the coefficient calculation unit 1035 calculates a coefficientset for predicting the pixel values by non-linear prediction.

[0959] In Step S1029, the coefficient calculation unit 1035 outputs thecoefficient set for each class, corresponding to the backgroundcomponent image to the coefficient set memory 1007, and the processingends.

[0960] As described above, the learning unit 1006-1 can generate thecoefficient set corresponding to the background component image.

[0961] The processing for generating of the coefficient setcorresponding to the foreground component image by the learning unit1006-2 corresponding to Step S1003 is the same as the processingdescribed with reference to the flowchart shown in FIG. 103 except forusing the foreground component image stored in the foreground componenttutor image frame memory 1002 and the student image corresponding to theforeground component image stored in the foreground component studentimage frame memory 105, and accordingly, description thereof will beomitted.

[0962] As described above, the movement-blurring-removed-imageprocessing unit 108 of which the configuration is shown in FIG. 96 cangenerate a coefficient set corresponding to the background componentimage which has been subjected to correction and a coefficient setcorresponding to the foreground component image which has been removalof movement blurring individually.

[0963]FIG. 104 is a block diagram which illustrates the configuration ofthe movement-blurring-removed-image processing unit 108 for generatingan even higher resolution image in the spatial direction by performingthe class classification adaptation processing. For example, themovement blurring removal processing unit 108 of which the configurationis shown in FIG. 104 generates an HD image by performing the classclassification adaptation processing based upon the input image which isa SD image.

[0964] Background component image frame memory 1101 stores thebackground component image which has been subjected to correctionsupplied from the correction unit 107. The background component imageframe memory 1101 supplies the stored background component image to amapping unit 1103-1.

[0965] Foreground component image frame memory 1102 stores theforeground component image which has been subjected to removal ofmovement blurring supplied from the movement blurring removal unit 106.The foreground component image frame memory 1102 supplies the storedforeground component image to a mapping unit 1103-2.

[0966] The mapping unit 1103-1 generates a predicted image correspondingto the background component image stored in the background componentimage frame memory 1101 by the class classification adaptationprocessing based upon the coefficient set corresponding to thebackground component image stored in the coefficient set memory 1104.The mapping unit 1103-1 supplies the generated predicted image to acorrection unit 1105.

[0967] The correction unit 1105 sets the pixel value of thepredetermined pixel in the predicted image corresponding to the mixedregion in the background component image corresponding to the movementblurring, which the movement blurring addition unit 1106 adds, to 0; ordivides the pixel value of the predetermined pixel in the predictedimage by the predetermined value corresponding to the movement blurringwhich is added. The correction unit 1005 supplies the predicted imagewhich has been subjected to correction described above to a synthesizingunit 1107.

[0968] The mapping unit 1103-2 generates a predicted image correspondingto the foreground component image stored in the foreground componentimage frame memory 1102 by the class classification adaptationprocessing based upon the coefficient set corresponding to theforeground component image stored in the coefficient set memory 1104.The mapping unit 1103-2 supplies the generated predicted image to themovement blurring addition unit 1106.

[0969] The movement blurring addition unit 1106 adds movement blurringto the predicted image by providing the desired movement blurringadjustment amount v′, e.g., the movement blurring adjustment amount v′of which value is the half value of the movement amount v of the inputimage or the movement blurring adjustment amount v′ having norelationship with the movement amount v. The movement blurring additionunit 1106 calculates the foreground component Fi/v′ by dividing thepixel value Fi in the predicted image in the foreground component imagewhich has subjected to removal of movement blurring by the movementblurring adjustment amount v′, calculates the sum of the foregroundcomponents Fi/v′s, and generates the pixel value which movement blurringis added to.

[0970] For example, in the event that the predicted image shown in FIG.105 is input, and the movement blurring adjustment amount v′ is 3, thepixel value C02 is (F01)/v′, the pixel value C03 is (F01+F02)/v′, thepixel value C04 is (F01+F02+F03)/v′, and the pixel value C05 is(F02+F03+F04)/v′ as shown in FIG. 106.

[0971] The movement blurring addition unit 1106 supplies the predictedimage of the foreground component image which has been subjected toaddition of movement blurring described above, to the synthesizing unit1107.

[0972] The synthesizing unit 1107 synthesizes the predicted imagecorresponding to the background component image which has been subjectedto correction supplied from the correction unit 1105, and the predictedimage corresponding to the foreground component image which has beensubjected to addition of movement blurring supplied from the movementblurring addition unit 1106, and supplies synthesized predicted image tothe frame memory 1108.

[0973] The frame memory 1108 stores the predicted image supplied fromthe synthesizing unit 1107, and also outputs the stored image as anoutput image.

[0974] In the event that there is no need to differentiate the mappingunit 1103-1 and the mapping unit 1103-2 individually, these will besimply referred to as the mapping unit 1103 below.

[0975]FIG. 107 is a block diagram which illustrates the configuration ofthe mapping unit 1103.

[0976] The mapping unit 1131 comprises a class classification unit 1141for performing class classification processing, and a prediction tapobtaining unit 1142 and a prediction computation unit 1143 forperforming the adaptation processing.

[0977] The class classification unit 1141 comprises a class tapobtaining unit 1151 and a waveform classification unit 1152, andperforms class classification for pixel of interest in the input imageof either background component image or foreground component image.

[0978] The class tap obtaining unit 1151 obtains a predetermined numberof class taps corresponding to pixel of interest in the input image, andsupplies the obtained class taps to the waveform classification unit1152. For example, the class tap obtaining unit 1151 obtains nine classtaps, and supplies the obtained class taps to the waveformclassification unit 1152.

[0979] The waveform classification unit 1152 reduces the number of bitsof the pixels making up the class taps by applying the ADRC processingto the class taps, classifies the pixel of interest into one of thepredetermined number of classes, and supplies the class No.corresponding to the classified class to the prediction tap obtainingunit 1142. For example, the waveform classification unit 1152 classifiesthe pixel of interest to one of 512 classes, and supplies the class No.corresponding to the classified class to the prediction tap obtainingunit 1142.

[0980] The prediction tap obtaining unit 1142 obtains the predeterminednumber of prediction taps corresponding to the class from the inputimage based upon the class No., and supplies the obtained predictiontaps and class No. to the prediction computation unit 1143.

[0981] The prediction computation unit 1143 obtains the coefficient setcorresponding to the class, and corresponding to the input image, fromthe coefficient set corresponding to the background component image andcoefficient set corresponding to the foreground component image, storedin the coefficient set memory 1104 based upon the class No. Theprediction computation unit 1143 predicts a pixel value in the predictedimage by linear prediction based upon the coefficient set and theprediction taps corresponding to the class, and corresponding to theinput image. The prediction computation unit 1143 supplies the predictedpixel value to the frame memory 1132.

[0982] Note that an arrangement may be made wherein the predictioncomputation unit 1143 predicts the pixel value in the predicted image bynon-linear prediction.

[0983] The frame memory 1132 stores the predicted pixel values suppliedfrom the mapping processing unit 1131, and outputs the image made up ofthe predicted pixel values.

[0984] Referring to the flowchart shown in FIG. 108, the processing forcreation of the image by the movement-blurring-removed-image processingunit 108 of which configuration is shown in FIG. 104 will be nowdescribed.

[0985] In Step S1101, the mapping unit 1103-1 predicts the imagecorresponding to the background component image stored in the backgroundcomponent image frame memory 1101 by the class classification adaptationprocessing based upon the coefficient set corresponding to thebackground component image stored in the coefficient set memory 1104.Details of the processing for prediction of the image corresponding tothe background component image will be described later with reference tothe flowchart shown in FIG. 109.

[0986] In Step S1102, the mapping unit 1103-2 predicts the imagecorresponding to the foreground component image stored in the foregroundcomponent image frame memory 1102 by the class classification adaptationprocessing based upon the coefficient set corresponding to theforeground component image stored in the coefficient set memory 1104.

[0987] In Step S1103, the correction unit 1105 corrects the predictedimage corresponding to the background component image.

[0988] In Step S1104, the movement blurring addition unit 1106 addsmovement blurring to the predicted image corresponding to the foregroundcomponent image.

[0989] In Step S1105, the synthesizing unit 1107 synthesizes thepredicted image corresponding to the background component image with thepredicted image corresponding to the foreground region. The synthesizingunit 1107 supplies the synthesized image to the frame memory 1108. Theframe memory 1108 stores the image supplied from the synthesizing unit1107.

[0990] In Step S1106, the frame memory 1108 outputs the stored andsynthesized image, and the processing ends.

[0991] As described above, the image processing device having themovement-blurring-removed-image processing unit 108 of whichconfiguration is shown in FIG. 104 generates a predicted imagecorresponding to the background component image and a predicted imagecorresponding to the foreground component image which has been subjectedto removal of movement blurring individually.

[0992] Note that it is needless to say that the processing in Step S1101and the processing in Step S1102 may be performed in a serial manner, aswell as in a parallel manner.

[0993] Referring to the flowchart shown in FIG. 109, the processing forprediction of the image corresponding to the background component imageby the mapping unit 1103-1 corresponding to Step S1101 will bedescribed.

[0994] In Step S1121, the mapping unit 1103-1 judges whether or notthere are any unprocessed pixels in the background component image, andin the event that judgment is made that there are unprocessed pixels inthe background component image, the flow proceeds to Step S1122, and themapping processing unit 1131 obtains the coefficient set correspondingto the background component image stored in the coefficient set memory1104. In Step S1123, the mapping processing unit 1131 obtains a pixel ofinterest from the background component image stored in the backgroundcomponent image frame memory 1101 in raster scan sequence.

[0995] In Step S1124, the class tap obtaining unit 1151 of the classclassification unit 1141 obtains the class tap corresponding to thepixel of interest from the background component image stored in thebackground component image frame memory 1101. In Step S1125, thewaveform classification unit 1152 of the class classification unit 1141reduces the number of bits of pixels making up the class tap by applyingthe ADRC processing to the class tap, and performs class classificationfor the pixel of interest. In Step S1126, the predication tap obtainingunit 1142 obtains the prediction tap corresponding to the pixel ofinterest from the background component image stored in the backgroundcomponent image frame memory 1101 based upon the classified class.

[0996] In Step S1127, the prediction computation unit 1143 predictspixel values of predicted image by linear prediction based upon thecoefficient set and the prediction tap, corresponding to the backgroundcomponent image and the classified class.

[0997] Note that the prediction computation unit 1143 may predict thepixel values of the predicted image by non-linear prediction, as well asby linear prediction.

[0998] In Step S1128, the prediction computation unit 1143 outputs thepredicted pixel value to the frame memory 1132. The frame memory 1132stores the pixel value supplied from the prediction computation unit1143. The procedure returns to Step S1121, and judgment whether or notany unprocessed pixels exist is repeated.

[0999] In Step S1121, in the event that judgment is made that there areno unprocessed pixels in the background component image, the flowproceeds to Step S1129, the frame memory 1132 outputs the storedpredicted image corresponding to the background component image, andprocessing ends.

[1000] As described above, the mapping unit 1103-1 can predict the imagecorresponding to the background component image based upon the correctedbackground component image.

[1001] The processing for generating of the predicted imagecorresponding to the foreground component image by the mapping unit1103-2 corresponding to Step S1102 is the same as the processingdescribed with reference to the flowchart shown in FIG. 109 except forusing the foreground component image stored in the foreground componentimage frame memory 1102 and the coefficient set corresponding to theforeground component image, and accordingly, description thereof will beomitted.

[1002] As described above, the movement-blurring-removed-imageprocessing unit 108 of which configuration is shown in FIG. 104 cangenerate a predicted image corresponding to the background componentimage and a predicted image corresponding to the foreground componentimage which has been subjected to removal of movement blurringindividually.

[1003]FIG. 110 is a flowchart which describes another processing forimage by the image processing device according to the present invention.In processing described with reference to the flowchart shown in FIG.110, the processing for edge enhancement is applied to the image whichhas been subjected to removal of movement blurring.

[1004] In Step S1201, the region specifying unit 103 specifies theforeground region, background region, covered background region, anduncovered background region, based upon the movement vector and theposition information supplied from the movement detecting unit 102, andthe input image. The processing in Step S1201 is the same as theprocessing in Step S101, and accordingly detailed description of theprocessing will be omitted.

[1005] In Step S1202, the mixture ratio calculation unit 104 calculatesthe mixture ratio α based upon the region information supplied from theregion specifying unit 103 and the input image. The processing in StepS1202 is the same as the processing in Step S102, so detaileddescription with regard to the processing will be omitted.

[1006] In Step S1203, the foreground/background separation unit 105separates the input image into the image in the foreground region, theimage in the background region, the foreground component image in thecovered background region, the background component image in the coveredbackground region, the foreground component image in the uncoveredbackground region, and the background component image in the uncoveredbackground region, based upon the region information supplied from theregion specifying unit 103 and the mixture ratio α supplied from themixture ratio calculation unit 104. The processing in Step S1203 is thesame as the processing in Step S103, and accordingly, descriptionthereof will be omitted.

[1007] In Step S1204, the movement blurring removal unit 106 removesmovement blurring from the foreground component image supplied from theforeground/background separation unit 105 based upon the movement vectorand the position information thereof, supplied from the movementdetecting unit 102, and the region information supplied from the regionspecifying unit 103. The processing in Step S1204 is the same as theprocessing in Step S104, and accordingly detailed description of theprocessing will be omitted.

[1008] In Step S1205, the correction unit 107 corrects pixel valuescorresponding to the mixed region in the background component imagesupplied from the foreground/background separation unit 105. Theprocessing in Step S1205 is the same as the processing in Step S105, andaccordingly, description thereof will be omitted.

[1009] In Step S1206, the movement-blurring-removed-image processingunit 108 performs the processing of edge enhancement for the foregroundcomponent image which has been subjected to removal of movementblurring, and the corrected background component image, respectively,and the processing ends. Details of the processing of edge enhancementperformed by the movement-blurring-removed-image processing unit 108will be described later.

[1010] As described above, the image processing device according to thepresent invention separates the input image into the foregroundcomponent image and the background component image, removes movementblurring from the foreground component image, and performs imageprocessing for the foreground component image which has been subjectedto removal of movement blurring, and the background component image,respectively.

[1011]FIG. 111 is a block diagram which illustrates the configuration ofthe movement-blurring-removed-image processing unit 108 for applyingedge enhancement processing with different effects for each backgroundcomponent image, or each foreground component image.

[1012] Background component image frame memory 1201 stores the correctedbackground component image supplied from the correction unit 107. Thebackground component image frame memory 1201 supplies the storedbackground component image to an edge enhancing unit 1203-1.

[1013] Foreground component image frame memory 1202 stores theforeground component image which has been subjected to removal ofmovement blurring, supplied from the movement blurring removal unit 106.The foreground component image frame memory 1202 supplies the storedforeground component image to an edge enhancing unit 1203-2.

[1014] The edge enhancing unit 1203-1 applies the processing of edgeenhancement suitable for the background component image to thebackground component image stored in the background component imageframe memory 1201.

[1015] For example, the edge enhancing unit 1203-1 performs theprocessing of edge enhancement which further enhances the edge for thebackground component image which is a still image as compared with theforeground component image. Thus the sense-of-resolution of thebackground component image can be improved without unnatural degradationof the image occurring in the event of applying the processing of edgeenhancement to images containing noise.

[1016] The edge enhancing unit 1203-1 supplies the background componentimage which has been subjected to edge enhancement to a correction unit1204.

[1017] The correction unit 1204 sets the pixel value of pixel in themixed region in the background component image to 0, or divides thepixel value of the pixel in the mixed region by the predetermined valuecorresponding to the movement blurring which is to be added,corresponding to the movement blurring added by a movement blurringaddition unit 1205. The correction unit 1204 supplies the imagecorrected as described above, to a synthesizing unit 1206.

[1018] The edge enhancing unit 1203-2 applies the processing of edgeenhancement suitable for the foreground component image, to theforeground component image stored in the foreground component imageframe memory 1202.

[1019] For example, the edge enhancing unit 1203-2 compares theforeground component image with the background component image, andperforms the processing of edge enhancement of which degree is less thanthat for the background component image. Thus the unnatural degradationin the image can be reduced as well as improving the sense-of-resolutionin the foreground component image even if the foreground component imagewhich has been subjected to removal of movement blurring contains noise.

[1020] The edge enhancing unit 1203-2 supplies the foreground componentimage which has been subjected to edge enhancement to the movementblurring addition unit 1205.

[1021] In the event that there is no need to differentiate the edgeenhancing unit 1203-1 and the edge enhancing unit 1203-2 individually,these will be referred to as the edge enhancing unit 1203 below.

[1022]FIG. 112 is a block diagram which illustrates the configuration ofthe edge enhancing unit 1203. The input image which is one of theforeground component image and the background component image, is inputto a high pass filter 1221 and an addition unit 1223.

[1023] The high pass filter 1221 extracts the components wherein thepixel value changes drastically with regard to pixel position, i.e., thehigh image frequency components from the input image based upon theinput filter coefficients, and removes the components wherein the changeof the pixel value is small with regard to the pixel position, i.e., thelow image frequency components, and generates an edge image.

[1024] For example, in the event of inputting the image shown in FIG.113A, the high pass filter 1221 generates the edge image shown in FIG.113B.

[1025] In the event that the input filter coefficients change, the highpass filter 1221 changes the image frequencies which are to beextracted, the image frequencies which are to be removed, and the gainfor the image which is to be extracted.

[1026] Referring to FIG. 114 through FIG. 117, the relationship betweenthe filter coefficients and the edge image will be described.

[1027]FIG. 114 is a diagram which illustrates the first example of thefilter coefficients. In FIG. 114, E indicates the exponent of 10. Forexample, E-04 indicates 10 ⁻⁴, and E-02 indicates 10⁻².

[1028] For example, the high pass filter 1221 multiplies each of pixelvalues, i.e., the pixel value of the pixel of interest, the pixel valuesof the pixels distanced from the pixel of interest by 1 pixel to 15pixels in a predetermined direction in the spatial direction Y, and thepixel values of pixels distanced from the pixel of interest by 1 pixelto 15 pixels in another direction in the spatial direction Y, by thecorresponding coefficient of the filter coefficients shown in FIG. 114.The high pass filter 1221 calculates the sum of the results obtained bymultiplying each pixel value of the pixels by the coefficientcorresponding thereto, and sets the calculated sum for the pixel valueof the pixel of interest.

[1029] For example, in the event of using the filter coefficients shownin FIG. 114, the high pass filter 1221 multiplies the pixel value of thepixel of interest by 1.2169396, multiplies the pixel value of the pixeldistanced from the pixel of interest by 1 pixel in the upper directionin the screen by −0.52530356, and multiplies the pixel value of thepixel distanced from the pixel of interest by 2 pixels in the upperdirection in the screen by −0.22739914.

[1030] In the same way, in the event of using the filter coefficientsshown in FIG. 114, the high pass filter 1221 multiplies each of pixelsdistanced from the pixel of interest by 3 pixels to 13 pixels in theupper direction in the screen by the corresponding coefficient,multiplies the pixel value of the pixel distanced from the pixel ofinterest by 14 pixels in the upper direction in the screen by−0.00022540586, and multiplies the pixel value of the pixel distancedfrom the pixel of interest by 15 pixels in the upper direction in thescreen by −0.00039273163.

[1031] In the event of using the filter coefficients shown in FIG. 114,in the same way, the high pass filter 1221 multiplies each of pixelsdistanced from the pixel of interest by 1 pixel to 15 pixels in thebottom direction in the screen by the corresponding coefficient.

[1032] The high pass filter 1221 calculates the sum of results obtainedby multiplying the pixel value of the pixel of interest, each pixelvalue of pixels distanced from the pixel of interest by 1 pixel to 15pixels in the top direction in the screen, and each pixel value ofpixels distanced from the pixel of interest by 1 pixel to 15 pixels inthe bottom direction in the screen, by the corresponding coefficient.The high pass filter 1221 sets the calculated sum to the pixel value ofthe pixel of interest.

[1033] The high pass filter 1221 moves the position of the pixel ofinterest in sequence in the spatial direction X, repeats theabove-described processing, and calculates pixel values for the entirescreen.

[1034] The high pass filter 1221 then multiplies the pixel value of theinterest, each pixel value of pixels distanced from the pixel ofinterest by 1 pixel to 15 pixels in a predetermined direction in thespatial direction X, and each pixel value of pixels distanced from thepixel of interest by 1 pixel to 15 pixels in another direction in thespatial direction X, in the image of which pixel values are calculatedbased upon the coefficients described above, by the correspondingcoefficient. The high pass filter 1221 calculates the sum of the resultsobtained by multiplying each pixel values of pixels by the correspondingcoefficient, and sets the calculated sum to the pixel value of the pixelof interest.

[1035] The high pass filter 1221 moves the position of the pixel ofinterest in sequence in the spatial direction Y, repeats theabove-described processing, and calculates pixel values of pixels forthe entire image.

[1036] That is to say, in this case, the high pass filter 1221 is aso-called one-dimensional filter using the coefficients shown in FIG.114.

[1037]FIG. 115 is a diagram which illustrates the operation of the highpass filter 1221 in the event of using the coefficients shown in FIG.114. As shown in FIG. 115, the maximum gain for the extracted imagecomponent at the high pass filter 1221 is 1 in the event of using thecoefficients shown in FIG. 114.

[1038]FIG. 116 is a diagram which illustrates the second example of thefilter coefficients.

[1039]FIG. 117 is a diagram which illustrates the operation of the highpass filter 1221 in the event that the same processing as the processingusing the filter coefficients shown in FIG. 114, is performed using thecoefficients shown in FIG. 116. As shown in FIG. 117, in the event ofusing the coefficients shown in FIG. 116, the maximum gain for extractedimage component at the high pass filter 1221 is 1.5.

[1040] As described above, the high pass filter 1221 changes the gainfor the extracted image component by the supplied filter coefficients.

[1041] While examples are not shown here, in the event of supplyingdifferent filter coefficients, the high pass filter 1221 can change theextracted image frequencies and the removed image frequencies in thesame way.

[1042] Returning to FIG. 112, the high pass filter 1221 supplies thegenerated edge image to a gain adjustment unit 1222.

[1043] The gain adjustment unit 1222 amplifies or reduces the edge imagesupplied from the high pass filter 1221 based upon the input gainadjustment coefficients. In the event that the input gain adjustmentcoefficient is altered, the gain adjustment unit 1222 changes theamplification ratio (or decay ratio) of the edge image. For example, inthe event of inputting the gain adjustment coefficients designating anamplification ratio which is equal to or more than 1, the gainadjustment unit 1222 amplifies the edge image, and in the event ofinputting the gain adjustment coefficients designating the amplificationratio which is less than 1, the gain adjustment unit 1222 attenuates theedge image.

[1044] The gain adjustment unit 1222 supplies the edge image which hasbeen subjected to gain adjustment to the addition unit 1223.

[1045] The addition unit 1223 adds the divided input image and the edgeimage which has been subjected to gain adjustment supplied from the gainadjustment unit 1222, and outputs the added image.

[1046] For example, in the event of inputting the input image shown inFIG. 113A, and supplying the edge image shown in FIG. 113B from the highpass filter 1221, the addition unit 1223 adds the input image shown inFIG. 113A and the edge image shown in FIG. 113B, and outputs the imageshown in FIG. 113C.

[1047] As described above, the edge enhancing unit 1203 applies the edgeenhancement processing for the input image.

[1048] For example, the edge enhancing unit 1203-1 of whichconfiguration is shown in FIG. 112 applies the edge enhancementprocessing, of which degree is even higher, to the background componentimage using the coefficients shown in FIG. 116. The edge enhancing unit1203-2 of which configuration is shown in FIG. 112 applies the edgeenhancement processing, of which degree is relatively lower, to theforeground component image using the coefficients shown in FIG. 114.

[1049]FIG. 118 is a block diagram which illustrates anotherconfiguration of the edge enhancing unit 1203. In the example shown inFIG. 118, edge enhancing unit 1203 comprises a filter 1241.

[1050] The filter 1241 generates an edge enhancement image by amplifyingthe components wherein the pixel value changes drastically with regardto the pixel position, i.e., the high image frequency components in theinput image, based upon the input filter coefficients.

[1051] For example, in the event of supplying the coefficients shown byway of an example in FIG. 119, the filter 1241 performs the sameprocessing as the processing described with regard to the high passfilter 1221, based upon the coefficients shown by way of an example inFIG. 119.

[1052]FIG. 120 is a diagram which illustrates the operation of thefilter 1241 in the event of using the coefficients shown in FIG. 119. Asshown in FIG. 120, in the event of using the coefficients shown in FIG.119, the filter 1241 amplifies the high image frequency components totwice, allows the low image frequency components to pass as they are,and generates an edge enhancement image.

[1053] In the event of using the coefficients shown in FIG. 119, thefilter 1241 outputs the same output image as the output image from theedge enhancing unit 1203 of which configuration is shown in FIG. 112 inthe event that the coefficients shown in FIG. 114 are used and the gainof the gain adjustment unit 1222 is 1.

[1054]FIG. 121 is a diagram which illustrates the second example of thefilter coefficients supplied to the filter 1241.

[1055]FIG. 122 is a diagram which illustrates the operation of thefilter 1241 in the event of using the coefficients shown in FIG. 121. Asshown in FIG. 122, in the event of using the coefficients shown in FIG.121, the filter 1241 amplifies the high image frequency components to2.5 times, allows the low image frequency components to pass as theyare, and generates an edge enhancement image.

[1056] In the event of using the coefficients shown in FIG. 121, thefilter 1241 outputs the same output image as the output image from theedge enhancing unit 1203 of which configuration is shown in FIG. 112 inthe event that the coefficients shown in FIG. 116 are used and the gainof the gain adjustment unit 1222 is 1.

[1057] As described above, the edge enhancing unit 1203 of whichconfiguration is shown in FIG. 118 can change the degree of edgeenhancement in the image by altering the gain of the high frequencycomponents in the image, by the input filter coefficients.

[1058] For example, the edge enhancing unit 1203-1 of whichconfiguration is shown in FIG. 118 applies the edge enhancementprocessing of which degree is even higher, using the coefficients shownin FIG. 121, to the background component image. The edge enhancing unit1203-1 of which configuration is shown in FIG. 118 applies the edgeenhancement processing of which degree is relatively lower, using thecoefficients shown in FIG. 119, to the foreground component image.

[1059] As described above, the edge enhancing unit 1203-1 and the edgeenhancing unit 1203-2 applies the edge enhancement processingcorresponding to the nature of the foreground component image or thebackground component image, to each foreground component image or eachbackground component image, based upon the different filter coefficientsor the gain adjustment coefficients, for example.

[1060] Returning to FIG. 111, the movement blurring addition unit 1205adds movement blurring to the image by giving a desired movementblurring adjustment amount v′, for example, the movement blurringadjustment amount v′ of which value is the half value of the movementamount v of the input image, or the movement blurring adjustment amountv′ of which value has no relationship with the movement amount v, by thesame processing as the movement blurring addition unit 1106. Themovement blurring addition unit 1205 calculates the foreground componentFi/v′ by dividing the pixel value in the foreground component imagewhich has been subjected to removal of movement blurring, Fi, by themovement blurring adjustment amount v′, calculates the sum of theforeground components Fi/v′s, and generates a pixel value which has beensubjected to addition of movement blurring.

[1061] The movement blurring addition unit 1205 adds movement blurringto the foreground component image which has been subjected to edgeenhancement, and supplies the foreground component image which has beensubjected to addition of movement blurring to a synthesizing unit 1206.

[1062] The synthesizing unit 1206 synthesizes the background componentimage which has been subjected to edge enhancement and correction,supplied from the correction unit 1204, with the foreground componentimage which has been subjected to edge enhancement and addition ofmovement blurring, supplied from the movement blurring addition unit1205, and supplies the synthesized image to frame memory 1207.

[1063] The frame memory 1207 stores the synthesized image supplied fromthe synthesizing unit 1206, and also outputs the stored image as anoutput image.

[1064] As described above, the movement-blurring-removed-imageprocessing unit 108 of which configuration is shown in FIG. 111 appliesthe edge enhancement processing corresponding to the nature of eachimage, for each background component image or each foreground componentimage, and accordingly the sense-of-resolution of the image is improvedwithout degrading the image unnaturally.

[1065]FIG. 123 is a diagram which describes the processing in themovement-blurring-removed-image processing unit 108 of whichconfiguration is shown in FIG. 111.

[1066] The specifying unit 103 specifies the foreground region,uncovered background region, covered background region, and backgroundregion in the input image. The input image of which regions arespecified, is separated into the background component image andforeground component image by the foreground/background separation unit105.

[1067] The movement blurring removal unit 106 removes movement blurringfrom the separated foreground component image. The correction unit 107corrects pixel values of the pixels corresponding to the mixed region inthe separated background component image.

[1068] The movement-blurring-removed-image processing unit 108 of whichconfiguration is shown in FIG. 111 performs edge enhancement for each ofthe corrected background component image and the foreground componentimage which has been subjected to removal of movement blurring,corresponding to the nature of each image.

[1069] The background component image which has been subjected to edgeenhancement is corrected, corresponding to addition of the movementblurring to the foreground component image. The desired movementblurring is added to the foreground component image which has beensubjected to edge enhancement.

[1070] The background component image which has been subjected to edgeenhancement and correction, and the foreground component image which hasbeen subjected to edge enhancement and addition of movement blurring,are synthesized.

[1071] A description will now be made with regard to the edgeenhancement processing in the movement-blurring-removed-image processingunit 108 corresponding to Step S1206 shown in FIG. 110, with referenceto the flowchart shown in FIG. 124.

[1072] In Step S1401, the edge enhancing unit 1203-1 performs edgeenhancement for the background component image stored in the backgroundcomponent image frame memory 1201 by edge enhancement processingcorresponding to the nature of the background component image.

[1073] In Step S1402, the edge enhancing unit 1203-2 performs edgeenhancement for the foreground component image stored in the foregroundcomponent image frame memory 1202 by the edge enhancement processingcorresponding to the nature of the foreground component image.

[1074] In Step S1403, the correction unit 1204 corrects pixel values inthe background component image corresponding to addition of the movementblurring to the foreground component image.

[1075] In Step S1404, the movement blurring addition unit 1205 adds thedesired movement blurring to the foreground component image.

[1076] In Step S1405, the synthesizing unit 1206 synthesizes thebackground component image which has been subjected to edge enhancementand correction, with the foreground component image which has beensubjected to edge enhancement and addition of movement blurring. Thesynthesizing unit 1206 supplies the synthesized image to the framememory 1207. The frame memory 1207 stores the image supplied from thesynthesizing unit 1206.

[1077] In Step S1406, the frame memory 1207 outputs the stored andsynthesized image, and the processing ends.

[1078] As described above, the movement-blurring-removed-imageprocessing unit 108 of which configuration is shown in FIG. 111 canperform the edge enhancement processing for each background componentimage and each foreground component image corresponding to the nature ofeach, and accordingly the sense-of-resolution can be improved withoutunnatural degradation in the image occurring.

[1079] Note that it is needless to say that the processing in Step S1401and Step S1402 may be performed serially or in parallel.

[1080]FIG. 125 is a block diagram which indicates the configuration ofthe movement-blurring-removed-image processing unit 108 for generating acoefficient set which is used for class classification adaptationprocessing for removing noise corresponding to the foreground componentimage which has been subjected to removal of movement blurring.

[1081] The movement-blurring-removed-image processing unit 108 of whichconfiguration is shown in FIG. 125 does not use the corrected backgroundcomponent image.

[1082] Tutor image frame memory 2201 stores the foreground componentimage which has been subjected to removal of movement blurring suppliedfrom the movement blurring removal unit 2001. The tutor image framememory 2201 supplies the foreground component image which has beensubjected to removal of movement blurring, which is the stored tutorimage, to a noise addition unit 2202 and a learning unit 2204.

[1083] The noise addition unit 2202 generates random numbers, and addsnoise to the foreground component image by adding the random numbers toeach pixel value in the foreground component image supplied from thetutor image frame memory 2201. The noise addition unit 2202 supplies theforeground component image which has been subjected to addition of noiseto a student image frame memory 2203.

[1084] The student image frame memory 2203 stores the student imagewhich is the foreground component image, which has been subjected toaddition of noise, supplied from the noise addition unit 2202. Thestudent image frame memory 2203 supplies the stored student image to thelearning unit 2204.

[1085] The learning unit 2204 generates an coefficient set correspondingto the foreground component image which has been subjected to additionof noise, based upon the tutor image which is the foreground componentimage supplied from the tutor image frame memory 2201 and the studentimage which is the foreground component image which has been subjectedto addition of noise supplied from the student image frame memory 2203,and supplies the generated coefficient set to a coefficient set memory2205.

[1086] The coefficient set memory 2205 stores the coefficient setcorresponding to the background component image supplied from thelearning unit 2204.

[1087] Referring to the flowchart shown in FIG. 126, description willnow be made with regard to the processing for learning for generating ofa coefficient set which is used for the class classification processingfor removing noise by the movement-blurring-removed-image processingunit 108 of which configuration is shown in FIG. 125.

[1088] In Step S2201, the noise addition unit 2202 generates a studentimage corresponding to the foreground component image which is the tutorimage by adding random numbers to the pixel values in the foregroundcomponent image which is the tutor image stored in the tutor image framememory 2201.

[1089] In Step S2202, the learning unit 2204 generates a coefficient setcorresponding to the foreground component image which has been subjectedto addition of noise, based upon the tutor image which is the foregroundcomponent image stored in the tutor image frame memory 2201 and thestudent image which is the foreground component image that has beensubjected to addition of noise stored in the student image frame memory2203, and supplies the generated coefficient set to the coefficient setmemory 2205. Details of the processing for generating a coefficient setis the same as the processing described with reference to the flowchartin FIG. 103, and accordingly description thereof will be omitted.

[1090] The coefficient set memory 2205 stores the coefficient setcorresponding to the foreground component image which has been subjectedto addition of noise, and the processing ends.

[1091] As described above, the movement-blurring-removed-imageprocessing unit 108 of which configuration is shown in FIG. 125 cangenerate a coefficient set corresponding to the foreground componentimage which has been subjected to addition of noise.

[1092]FIG. 127 is a block diagram which indicates the configuration ofthe movement-blurring-removed-image processing unit 108 for removingnoise and performing edge enhancement for the background component imageby performing the class classification adaptation processing for theforeground component image which has been subjected to removal ofmovement blurring.

[1093] Frame memory 2301 stores the foreground component image which hasbeen subjected to removal of movement blurring supplied from themovement blurring removal unit 2001. The frame memory 2301 supplies thestored foreground component image which has been subjected to removal ofmovement blurring to a mapping unit 2302.

[1094] The mapping unit 2302 generates a predicted image which has beensubjected to removal of noise corresponding to the foreground componentimage stored in the frame memory 2301 based upon the coefficient setcorresponding to the foreground component image stored in thecoefficient set memory 2303, by the class classification processing. Themapping unit 2302 supplies the generated predicted image to frame memory2304.

[1095] The frame memory 2304 stores the predicted image which has beensubjected to removal of noise, and supplies the stored predicted imageto a synthesizing unit 2308.

[1096] The frame memory 2305 stores the background component imagesupplied from the correction unit 2002. The frame memory 2305 suppliesthe stored background component image to an edge enhancing unit 2306.

[1097] The edge enhancing unit 2306 enhances the edge in the backgroundcomponent image stored in the frame memory 2305 by processing of edgeenhancement, and supplies the background component image which has beensubjected to edge enhancement to frame memory 2307.

[1098] The frame memory 2307 stores the background component image whichhas been subjected to edge enhancement, and supplies the storedbackground component image to the synthesizing unit 2308.

[1099] The synthesizing unit 2308 synthesizes the predicted image whichhas been subjected to removal of noise corresponding to the foregroundcomponent image supplied from the frame memory 2304, with the backgroundcomponent image which has been subjected to edge enhancement suppliedfrom the frame memory 2307, and outputs the synthesized image as anoutput image.

[1100]FIG. 128 is a diagram which describes the processing of themovement-blurring-removed-image processing unit 108.

[1101] As shown in FIG. 128, the input image is divided into regions,and separated into the foreground component image and the backgroundcomponent image. The separated input image is synthesized into theforeground component image and the background component image.

[1102] Movement blurring contained in the foreground component image isremoved. The background component image is corrected with regard to thepixel values corresponding to the mixed region.

[1103] The movement-blurring-removed-image processing unit 108 removesnoise in the foreground component image which has been subjected toremoval of movement blurring by applying the class classificationadaptation processing, and performs edge enhancement for the backgroundcomponent image which has been subjected to correction.

[1104] Referring to the flowchart shown in FIG. 129, the processing forcreation of the image by the movement-blurring-removed-image processingunit 108 of which configuration is shown in FIG. 127, will now bedescribed.

[1105] In Step S2301, the mapping unit 2302 predicts the image which hasbeen subjected to removal of noise from the foreground component imagestored in the frame memory 7301 by the class classification adaptationprocessing based upon the coefficient set corresponding to theforeground component image stored in the coefficient set memory 2303.Details of the processing for prediction of the image are the same asthe processing described with reference to the flowchart shown in FIG.109 except for using the foreground component image instead of thebackground component image, and accordingly description thereof will beomitted.

[1106] The mapping unit 2302 supplies the image wherein the noise hasbeen removed from the foreground component image, to the frame memory2304. The frame memory 2304 stores the predicted image which has beensubjected to removal of noise corresponding to the foreground componentimage, and supplies the stored predicted image to the synthesizing unit2308.

[1107] In Step S2302, the edge enhancing unit 2306 performs edgeenhancement processing for the background component image stored in theframe memory 2305. The edge enhancing unit 2306 supplies the image whichhas been subjected to edge enhancement to the frame memory 2307. Theframe memory 2307 stores the image which has been subjected to edgeenhancement, and supplies the stored edge-enhanced image to thesynthesizing unit 2308.

[1108] In Step S2303, the synthesizing unit 2308 synthesizes thepredicted image which has been subjected to removal of noisecorresponding to the foreground component image, and the edge-enhancedbackground component image. The synthesizing unit 2308 outputs thestored synthesized image, and the processing ends.

[1109] As described above, the image processing device having themovement-blurring-removed-image processing unit 108 of whichconfiguration is shown in FIG. 127 can generate a predicted image whichhas been subjected to removal of noise corresponding to the foregroundcomponent image, perform edge enhancement processing for the backgroundcomponent image, and output the image by synthesizing the predictedimage which has been subjected to removal of noise and the edge enhancedbackground component image, and accordingly noise in the foregroundcomponent image due to the processing for removal of movement blurringcan be reduced, and also the sense-of-resolution with regard to theentire image can be improved.

[1110] Note that it is needless to say that the processing in Step S2301and Step S2302 may be performed in serial, as well as in parallel.

[1111]FIG. 130 is a block diagram which illustrates anotherconfiguration of the functions of the image processing device. While theimage processing device shown in FIG. 11 performs the regionspecification and the calculation of the mixture ratio α in sequence,the image processing device shown in FIG. 130 performs the regionspecification and the calculation of mixture ratio α in a parallelmanner.

[1112] The same portions as the functions shown in the block diagram inFIG. 11 are denoted by the same reference numerals, and descriptionthereof will be omitted.

[1113] The input image is supplied to the objected extracting unit 101,the region specifying unit 103, a mixture ratio calculation unit 3001,and a foreground/background separation unit 3002.

[1114] The mixture ratio calculation unit 3001 calculates the estimatedmixture ratio wherein an assumption is made that the pixel belongs tothe covered background region, and the estimated mixture ratio whereinan assumption is made that the pixel belongs to the uncovered backgroundregion, for each of pixels contained in the input image based upon theinput image, and supplies the estimated mixture ratio wherein anassumption is made that the calculated pixel belongs to the coveredbackground region, and the estimated mixture ratio wherein an assumptionis made that the pixel belongs to the uncovered background region, tothe foreground/background separation unit 3002.

[1115]FIG. 131 is a block diagram which illustrates one example of theconfiguration of the mixture ratio calculation unit 3001.

[1116] The estimated mixture ratio processing unit 401 shown in FIG. 131is the same as the estimated mixture ratio processing unit 401 shown inFIG. 62. The estimated mixture ratio processing unit 402 shown in FIG.131 is the same as the estimated mixture ratio processing unit 402 shownin FIG. 62.

[1117] The estimated mixture ratio processing unit 401 calculates theestimated mixture ratio for each pixel by the computation correspondingto the model of the covered background region based upon the inputimage, and outputs the calculated estimated mixture ratio.

[1118] The estimated mixture ratio processing unit 402 calculates theestimated mixture ratio for each pixel by the computation correspondingto the model of the uncovered background region based upon the inputimage, and outputs the calculated estimated mixture ratio.

[1119] The foreground/background separation unit 3002 separates theinput image into the background component image and the foregroundcomponent image based upon the estimated mixture ratio wherein anassumption is made that the pixel belongs to the covered backgroundregion and the estimated mixture ratio wherein an assumption is madethat the pixel belongs to the uncovered background region, supplied fromthe mixture ratio calculation unit 3001, and the region informationsupplied from the region specifying unit 103, and supplies the separatedimages to the movement-blurring-removed-image processing unit 108.

[1120]FIG. 132 is a block diagram which illustrates one example of theconfiguration of the foreground/background separation unit 3002.

[1121] The same portions as the movement blurring removal unit 106 shownin FIG. 80 are denoted by the same reference numerals, and descriptionthereof will be omitted.

[1122] A selection unit 3021 selects either of the estimated mixtureratio wherein an assumption is made that the pixel belongs to thecovered background region, or the estimated mixture ratio wherein anassumption is made that the pixel belongs to the uncovered backgroundregion, supplied from the mixture ratio calculation unit 3001 based uponthe region information supplied from the region specifying unit 103, andsupplies the selected estimated mixture ratio as a mixture ratio α tothe separation unit 601.

[1123] The separation unit 601 extracts the foreground components andthe background components from the pixel values of the pixels belongingto the mixed region based upon the mixture ratio α supplied from theselection unit 3021 and the region information, and separates into thebackground components in the uncovered background region, the foregroundcomponents in the uncovered background region, the background componentsin the covered background region, and the foreground components in thecovered background region.

[1124] The configuration of the separation unit 601 may be the same asthe configuration shown in FIG. 85.

[1125] As described above, the image processing device of whichconfiguration is shown in FIG. 130 can perform processing for eachbackground component image and each foreground component image,corresponding to the nature of each image.

[1126] As described above, in the image processing device according tothe present invention, the input image is separated into the backgroundcomponent image and the foreground component image, and the processingsuitable for the separated images is performed, and accordingly an evenhigher resolution image is generated without unnatural images occurring,for example.

[1127]FIG. 133 is a block diagram which illustrates anotherconfiguration of the functions of the image processing device.

[1128] The same portions as those shown in FIG. 11 are denoted by thesame reference numerals, and description thereof will be omitted.

[1129] The object extracting unit 101 roughly extracts the image objectscorresponding to the foreground objects contained in the input image,and supplies the extracted image objects to the movement detecting unit102.

[1130] The movement detecting unit 102 calculates the movement vectorsof the image objects corresponding to the roughly extracted foregroundobjects by techniques such as block matching, gradation, phasecorrelation, and pixel recursion, or the like, and provides thecalculated movement vectors and movement vector position to the regionspecifying unit 103.

[1131] The region specifying unit 103 classifies each of pixels of theinput images into one of foreground region, background region, or mixedregion which consists of the covered background region and the uncoveredbackground region, and supplies the region information to the mixtureratio calculation unit 104 and a foreground/background separation unit4001.

[1132] The mixture ratio calculation unit 104 calculates the mixtureratio α corresponding to the pixels contained in the mixed region basedupon the input image and the region information supplied from the regionspecifying unit 103, and supplies the calculated mixture ratio to theforeground/background separation unit 4001.

[1133] The foreground/background separation unit 4001 separates theinput image into the components of the image corresponding to theforeground object and the background component image which consists ofonly the background components based upon the region informationsupplied from the region specifying unit 103 and the mixture ratio αsupplied from the mixture ratio calculation unit 104, and supplies theimage in the background region, the image which consists of only thebackground components in the uncovered background region (which will bereferred to as the background component image in the uncoveredbackground region below), the image which consists of only theforeground components in the uncovered background region (which will bereferred to as the foreground component image in the uncoveredbackground region below), the image which consists of only thebackground components in the covered background region (which will bereferred to as the background component image in the covered backgroundregion below), the image which consists of only the foregroundcomponents in the covered background region (which will be referred toas the foreground component image in the covered background regionbelow), and the image in the foreground region, to a separated imageprocessing unit 4002.

[1134] The separated image processing unit 4002 performs processing forthe image in the background region, the background component image inthe uncovered background region, the foreground component image in theuncovered background region, the background component image in thecovered background region, the foreground component image in the coveredbackground region, and the image in the foreground region, supplied fromthe foreground/background separation unit 4001, respectively.

[1135] For example, the separated image processing unit 4002 generatescoefficients which are used in the class classification adaptationprocessing for generating a even higher resolution image for each of theimage in the background region, background component image in theuncovered background region, foreground component image in the uncoveredbackground region, background component image in the covered backgroundregion, foreground component image in the covered background region, andimage in the foreground region.

[1136] For example, the separated image processing unit 4002 creates aneven higher resolution image by applying the class classificationadaptation processing for each of the image in the background region,background component image in the uncovered background region,foreground component image in the uncovered background region,background component image in the covered background region, foregroundcomponent image in the covered background region, and image in theforeground region.

[1137]FIG. 134 is a diagram which describes the processing in theseparated image processing unit 4002 of which configuration is shown inFIG. 133. The region specifying unit 103 specifies the foregroundregion, background region, covered background region, and uncoveredbackground region, in the input image.

[1138] The input image wherein the regions thereof are specified and themixture ratio α is detected by the mixture ratio calculation unit 104,is separated into the image in the foreground region, image in thebackground region, foreground component image in the covered backgroundregion, background component image in the covered background region,foreground component image in the uncovered background region, andbackground component image in the uncovered background region by theforeground/background separation unit 4001.

[1139] The separated image processing unit 4002 calculates a coefficientset corresponding to the image in the foreground region, a coefficientset corresponding to the image in the background region, a coefficientset corresponding to the foreground component image in the coveredbackground region, a coefficient set corresponding to the backgroundcomponent image in the covered background region, a coefficient setcorresponding to the foreground component image in the uncoveredbackground region, and a coefficient set corresponding to the backgroundcomponent image in the uncovered background region individually, basedupon the separated image in the foreground region, image in thebackground region, foreground component image in the covered backgroundregion, background component image in the covered background region,foreground component image in the uncovered background region, andbackground component image in the uncovered background region.

[1140] The coefficient set corresponding to the background region isused for prediction of the pixel value in the background region in theclass classification adaptation processing for predicting pixel values.The coefficient set corresponding to the background component image inthe uncovered background region is used for prediction of the pixelvalue corresponding to the background component image in the uncoveredbackground region in the class classification adaptation processing forpredicting pixel values. The coefficient set corresponding to theforeground component image in the uncovered background region is usedfor prediction of the pixel value corresponding to the foregroundcomponent image in the uncovered background region in the classclassification adaptation processing for predicting pixel values.

[1141] The coefficient set corresponding to the background componentimage in the covered background region is used for prediction of thepixel value corresponding to the background component image in thecovered background region in the class classification adaptationprocessing for predicting pixel values. The coefficient setcorresponding to the foreground component image in the coveredbackground region is used for prediction of the pixel valuecorresponding to the foreground component image in the coveredbackground region in the class classification adaptation processing forpredicting pixel values.

[1142] The coefficient set corresponding to the foreground region isused for prediction of the pixel value in the foreground region in theclass classification adaptation processing for predicting pixel values.

[1143] The predicted image corresponding to the image in the backgroundregion, the predicted image corresponding to the background componentimage in the uncovered background region, the predicted imagecorresponding to the foreground component image in the uncoveredbackground region, the predicted image corresponding to the backgroundcomponent image in the covered background region, the predicted imagecorresponding to the foreground component image in the coveredbackground region, and the predicted image corresponding to the image inthe foreground region, are synthesized into a single predicted image.

[1144]FIG. 135 is a block diagram which illustrates one example of theconfiguration of the foreground/background separation unit 4001. Theinput image supplied to the foreground/background separation unit 4001is supplied to a separation unit 4101, a switch 4102, and a switch 4103.The region information which indicates the covered background region andthe uncovered background region supplied from the region specifying unit103 is supplied to the separation unit 4101. The region informationwhich indicates the foreground region is supplied to the switch 4102.The region information which indicates the background region is suppliedto the switch 4103.

[1145] The mixture ratio α supplied from the mixture ratio calculationunit 104 is supplied to the separation unit 4101.

[1146] The separation unit 4101 separates the background components fromthe covered background region in the input image, as well as foregroundcomponents, based upon the region information which indicates thecovered background region and the mixture ratio α, and outputs theforeground component image in the covered background region whichconsists of the separated foreground components, and the backgroundcomponent image in the covered background region which consists of theseparated background components.

[1147] The separation unit 4101 separates the background components fromthe uncovered background region in the input image, as well asforeground components, based upon the region information which indicatesthe uncovered background region and the mixture ratio α, and outputs theforeground component image in the uncovered background region whichconsists of the separated foreground components, and the backgroundcomponent image in the uncovered background region which consists of theseparated background components.

[1148] In the event of inputting the pixel corresponding to theforeground region, the switch 4102 is closed based upon the regioninformation which indicates the foreground region, and outputs the imagein the foreground region.

[1149] In the event of inputting the pixel corresponding to thebackground region, the switch 4103 is closed based upon the regioninformation which indicates the background region, and outputs the imagein the background region.

[1150]FIG. 136 is a block diagram which illustrates one example of theconfiguration of the separation unit 4101. The image input to theseparation unit 4101 is supplied to frame memory 4121, and the regioninformation which indicates the covered background region and theuncovered background region and the mixture ratio α supplied from themixture ratio calculation unit 104 are input to a separation processingblock 4122.

[1151] The frame memory 4121 stores input image in increments of frames.In the event that the frame #n is the object of the processing, theframe memory 4121 stores the frame #n−1 one frame previous to the frame#n, the frame #n, and the frame #n+1 one frame following the frame #n.

[1152] The frame memory 4121 supplies pixels corresponding to the frame#n−1, the frame #n, and the frame #n+1 to the separation processingblock 4122.

[1153] The separation processing block 4122 separates the foregroundcomponents and the background components from the pixel belonging to themixed region in the frame #n by applying the calculation described withreference to FIG. 83 and FIG. 84 to the pixel values of the pixelscorresponding to the frame #n−1, the frame #n, and the frame #n+1supplied from the frame memory 4121 based upon the region informationwhich indicates the covered background region and the uncoveredbackground region and the mixture ratio α.

[1154] The separation processing block 4122 comprises an uncoveredregion processing unit 4131 and a covered region processing unit 4132.

[1155] A multiplication device 4141 of the uncovered region processingunit 4131 multiplies the pixel value of the pixel in the frame #n+1supplied from the frame memory 4121 by the mixture ratio α, and outputsto a switch 4142. In the event that the pixel (corresponding to thepixel in the frame #n+1) in the frame #n supplied from the frame memory4121 belongs to the uncovered background region, the switch 4142 isclosed, and supplies the pixel value multiplied by the mixture ratio αsupplied from the multiplication device 4141 to a computation device4143. The value wherein the pixel value of the pixel in the frame #n+1which is output from the switch 4142 multiplied by the mixture ratio α,is the same as the background component of the pixel value of the pixelcorresponding to the frame #n, which is output as a background componentimage in the uncovered background region.

[1156] The computation device 4143 obtains the foreground component bysubtracting the background component supplied from the switch 4142 fromthe pixel value of the pixel in the frame #n supplied from the framememory 4121. The computation device 4143 outputs the foregroundcomponent image made up of pixels in the frame #n belonging to theuncovered background region.

[1157] A multiplication device 4151 of the covered region processingunit 4132 multiplies the pixel value of the pixel in the frame #n−1supplied from the frame memory 4121 by the mixture ratio α, and outputsto a switch 4152. In the event that the pixel (corresponding to thepixel in the frame #n−1) in the frame #n supplied from the frame memory4121 belongs to the covered background region, the switch 4152 isclosed, and supplies the pixel value multiplied by the mixture ratio αsupplied from the multiplication device 4151 to a computation device4153. The value wherein the pixel value of the pixel in the frame #n−1which is output from the switch 4152 multiplied by the mixture ratio α,is the same as the background component of the pixel value of the pixelcorresponding to the frame #n, which is output as a background componentimage in the covered background region.

[1158] The computation device 4153 obtains the foreground component bysubtracting the background component supplied from the switch 4152 fromthe pixel value of the pixel in the frame #n supplied from the framememory 4121. The computation device 4153 outputs the foregroundcomponent image made up of pixels in the frame #n belonging to thecovered background region.

[1159] Using the mixture ratio α which is the amount of features enablesentire separation of the foreground component and background component,contained in the pixel value.

[1160]FIG. 137 is a block diagram which illustrates the configuration ofthe separated image processing unit 4002 for generating a coefficientset which is used for the class classification adaptation processing forgenerating a even higher resolution image in the spatial direction. Forexample, the separated image processing unit 4002 of which configurationis shown in FIG. 137 generates a coefficient set which is used for theclass classification adaptation processing for generating an HD imagefrom an SD image based upon the input HD image.

[1161] Background region tutor image frame memory 4201 stores the imagein the background region in the tutor image supplied from theforeground/background separation unit 4001. The background region tutorimage frame memory 4201 supplies the stored image in the backgroundregion in the tutor image to a weighted averaging unit 4207-1 and alearning unit 4214-1.

[1162] Uncovered background region background component tutor imageframe memory 4202 stores the background component image in the uncoveredbackground region in the tutor image supplied from theforeground/background separation unit 4001. The uncovered backgroundregion background component tutor image frame memory 4202 supplies thestored background component image in the uncovered background region inthe tutor image to a weighted averaging unit 4207-2 and a learning unit4214-2.

[1163] Uncovered background region foreground component tutor imageframe memory 4203 stores the foreground component image in the uncoveredbackground region in the tutor image supplied from theforeground/background separation unit 4001. The uncovered backgroundregion foreground component tutor image frame memory 4203 supplies thestored foreground component image in the uncovered background region inthe tutor image to a weighted averaging unit 4207-3 and a learning unit4214-3.

[1164] Covered background region background component tutor image framememory 4204 stores the background component image in the coveredbackground region in the tutor image supplied from theforeground/background separation unit 4001. The covered backgroundregion background component tutor image frame memory 4204 supplies thestored background component image in the covered background region inthe tutor image to a weighted averaging unit 4207-4 and a learning unit4214-4.

[1165] Covered background region foreground component tutor image framememory 4205 stores the foreground component image in the coveredbackground region in the tutor image supplied from theforeground/background separation unit 4001. The covered backgroundregion foreground component tutor image frame memory 4205 supplies thestored foreground component image in the covered background region inthe tutor image to a weighted averaging unit 4207-5 and a learning unit4214-5.

[1166] Foreground region tutor image frame memory 4206 stores the imagein the foreground region in the tutor image supplied from theforeground/background separation unit 4001. The foreground region tutorimage frame memory 4206 supplies the stored image in the foregroundregion in the tutor image to a weighted averaging unit 4207-6 and alearning unit 4214-6.

[1167] The weighted averaging unit 4207-1 generates an SD image which isa student image by one-quarter weighted-averaging the image in thebackground region in the tutor image which is an HD image, for example,supplied from the background region tutor image frame memory 4201, andsupplies the generated SD image to background region student image framememory 4208.

[1168] The background region student image frame memory 4208 stores thestudent image corresponding to the image in the background region in thetutor image supplied from the weighted averaging unit 4207-1. Thebackground region student image frame memory 4208 supplies the storedstudent image corresponding to the image in the background region in thetutor image to the learning unit 4214-1.

[1169] The weighted averaging unit 4207-2 generates an SD image which isa student image by one-quarter weighted-averaging the backgroundcomponent image in the uncovered background region in the tutor imagewhich is an HD image, for example, supplied from the uncoveredbackground region background component tutor image frame memory 4202,and supplies the generated SD image to uncovered background regionbackground component student image frame memory 4209.

[1170] The uncovered background region background component studentimage frame memory 4209 stores the student image, which is an SD image,corresponding to the background component image in the uncoveredbackground region in the tutor image supplied from the weightedaveraging unit 4207-2. The uncovered background region backgroundcomponent student image frame memory 4209 supplies the stored studentimage corresponding to the background component image in the uncoveredbackground region in the tutor image to the learning unit 4214-2.

[1171] The weighted averaging unit 4207-3 generates an SD image which isa student image by one-quarter weighted-averaging the foregroundcomponent image in the uncovered background region in the tutor imagewhich is an HD image, for example, supplied from the uncoveredbackground region foreground component tutor image frame memory 4203,and supplies the generated SD image to uncovered background regionforeground component student image frame memory 4210.

[1172] The uncovered background region foreground component studentimage frame memory 4210 stores the student image, which is an SD image,corresponding to the foreground component image in the uncoveredbackground region in the tutor image supplied from the weightedaveraging unit 4207-3. The uncovered background region foregroundcomponent student image frame memory 4210 supplies the stored studentimage corresponding to the foreground component image in the uncoveredbackground region in the tutor image to the learning unit 4214-3.

[1173] The weighted averaging unit 4207-4 generates an SD image, whichis a student image, by one-quarter weighted-averaging the backgroundcomponent image in the covered background region in the tutor image, forexample, supplied from the covered background region backgroundcomponent tutor image frame memory 4204, and supplies the generated SDimage to covered background region background component student imageframe memory 4211.

[1174] The covered background region background component student imageframe memory 4211 stores the student image, which is an SD image,corresponding to the background component image in the coveredbackground region in the tutor image supplied from the weightedaveraging unit 4207-4. The covered background region backgroundcomponent student image frame memory 4211 supplies the stored studentimage corresponding to the background component image in the coveredbackground region in the tutor image to the learning unit 4214-4.

[1175] The weighted averaging unit 4207-5 generates an SD image, whichis a student image, by one-quarter weighted-averaging the foregroundcomponent image in the covered background region in the tutor image, forexample, supplied from the covered background region foregroundcomponent tutor image frame memory 4205, and supplies the generated SDimage to covered background region foreground component student imageframe memory 4212.

[1176] The covered background region foreground component student imageframe memory 4212 stores the student image, which is an SD image,corresponding to the foreground component image in the coveredbackground region in the tutor image supplied from the weightedaveraging unit 4207-5. The covered background region foregroundcomponent student image frame memory 4212 supplies the stored studentimage corresponding to the foreground component image in the coveredbackground region in the tutor image to the learning unit 4214-5.

[1177] The weighted averaging unit 4207-6 generates an SD image which isa student image by one-quarter weighted-averaging the image in theforeground region in the tutor image which is an HD image, for example,supplied from the foreground region tutor image frame memory 4206, andsupplies the generated SD image to foreground region student image framememory 4213.

[1178] The foreground region student image frame memory 4213 stores thestudent image, which is an SD image, corresponding to the image in theforeground region in the tutor image supplied from the weightedaveraging unit 4207-6. The foreground region student image frame memory4213 supplies the stored student image corresponding to the image in theforeground region in the tutor image to the learning unit 4214-6.

[1179] The learning unit 4214-1 generates a coefficient setcorresponding to the background region based upon the image in thebackground region in the tutor image supplied from the background regiontutor image frame memory 4201 and the student image corresponding to theimage in the background region in the tutor image supplied from thebackground region student image frame memory 4208, and supplies thegenerated coefficient set to coefficient set memory 4215.

[1180] The learning unit 4214-2 generates a coefficient setcorresponding to the background component image in the uncoveredbackground region based upon the background component image in theuncovered background region in the tutor image supplied from theuncovered background region background component tutor image framememory 4202 and the student image corresponding to the backgroundcomponent image in the uncovered background region in the tutor imagesupplied from the uncovered background region background componentstudent image frame memory 4209, and supplies the generated coefficientset to the coefficient set memory 4215.

[1181] The learning unit 4214-3 generates a coefficient setcorresponding to the foreground component image in the uncoveredbackground region based upon the foreground component image in theuncovered background region in the tutor image supplied from theuncovered background region foreground component tutor image framememory 4203 and the student image corresponding to the foregroundcomponent image in the uncovered background region in the tutor imagesupplied from the uncovered background region foreground componentstudent image frame memory 4210, and supplies the generated coefficientset to the coefficient set memory 4215.

[1182] The learning unit 4214-4 generates a coefficient setcorresponding to the background component image in the coveredbackground region based upon the background component image in thecovered background region in the tutor image supplied from the coveredbackground region background component tutor image frame memory 4204 andthe student image corresponding to the background component image in thecovered background region in the tutor image supplied from the coveredbackground region background component student image frame memory 4211,and supplies the generated coefficient set to the coefficient set memory4215.

[1183] The learning unit 4214-5 generates a coefficient setcorresponding to the foreground component image in the coveredbackground region based upon the foreground component image in thecovered background region in the tutor image supplied from the coveredbackground region foreground component tutor image frame memory 4205 andthe student image corresponding to the foreground component image in thecovered background region in the tutor image supplied from the coveredbackground region foreground component student image frame memory 4212,and supplies the generated coefficient set to the coefficient set memory4215.

[1184] The learning unit 4214-6 generates a coefficient setcorresponding to the foreground region based upon the image in theforeground region in the tutor image supplied from the foreground regiontutor image frame memory 4206 and the student image corresponding to theimage in the foreground region in the tutor image supplied from theforeground region student image frame memory 4213, and supplies thegenerated coefficient set to the coefficient set memory 4215.

[1185] The coefficient set memory 4215 stores the coefficient setcorresponding to the background region supplied from the learning unit4214-1, the coefficient set corresponding to the background componentimage in the uncovered background region supplied from the learning unit4214-2, the coefficient set corresponding to the foreground componentimage in the uncovered background region supplied from the learning unit4214-3, the coefficient set corresponding to the background componentimage in the covered background region supplied from the learning unit4214-4, the coefficient set corresponding to the foreground componentimage in the covered background region supplied from the learning unit4214-5, and the coefficient set corresponding to the foreground regionsupplied from the learning unit 4214-6.

[1186] Note that the learning units 4214-1 through 4214-6 have the sameconfiguration as the learning unit 1006, so description thereof will beomitted.

[1187]FIG. 138 is a block diagram which illustrates the configuration ofthe separated image processing unit 4002 for generating an even higherresolution image in the spatial direction by performing the classclassification adaptation processing. For example, the separated imageprocessing unit 4002 of which configuration is shown in FIG. 138generates an HD image by performing the class classification processingbased upon the input image which is an SD image.

[1188] Background region frame memory 4301 stores the image in thebackground region made up of pixels belonging to the background regionsupplied from the foreground/background separation unit 4001. Thebackground region frame memory 4301 supplies the stored image in thebackground region to a mapping unit 4307-1.

[1189] Uncovered background region background component image framememory 4302 stores the background component image in the uncoveredbackground region supplied from the foreground/background separationunit 4001. The uncovered background region background component imageframe memory 4302 supplies the stored background component image in theuncovered background region to a mapping unit 4307-2.

[1190] Uncovered background region foreground component image framememory 4303 stores the foreground component image in the uncoveredbackground region supplied from the foreground/background separationunit 4001. The uncovered background region foreground component imageframe memory 4303 supplies the stored foreground component image in theuncovered background region to a mapping unit 4307-3.

[1191] Covered background region background component image frame memory4304 stores the background component image in the covered backgroundregion supplied from the foreground/background separation unit 4001. Thecovered background region background component image frame memory 4304supplies the stored background component image in the covered backgroundregion to a mapping unit 4307-4.

[1192] Covered background region foreground component image frame memory4305 stores the foreground component image in the covered backgroundregion supplied from the foreground/background separation unit 4001. Thecovered background region foreground component image frame memory 4305supplies the stored foreground component image in the covered backgroundregion to a mapping unit 4307-5.

[1193] Foreground region frame memory 4306 stores the image in theforeground region made up of pixels belonging to the foreground regionsupplied from the foreground/background separation unit 4001. Theforeground region image frame memory 2306 supplies the stored image inthe foreground region to a mapping unit 4307-6.

[1194] The mapping unit 4307-1 generates a predicted image correspondingto the image in the background region stored in the background regionframe memory 4301 by the class classification adaptation processingbased upon the coefficient set corresponding to the background regionstored in coefficient set memory 4308. The mapping unit 4307-1 suppliesthe generated predicted image to a synthesizing unit 4309.

[1195] The mapping unit 4307-2 generates a predicted image correspondingto the background component image in the uncovered background regionstored in the uncovered background region background component imageframe memory 4302 by the class classification adaptation processingbased upon the coefficient set corresponding to the background componentimage in the uncovered background region stored in the coefficient setmemory 4308. The mapping unit 4307-2 supplies the generated predictedimage to the synthesizing unit 4309.

[1196] The mapping unit 4307-3 generates a predicted image correspondingto the foreground component image in the uncovered background regionstored in the uncovered background region foreground component imageframe memory 4303 by the class classification adaptation processingbased upon the coefficient set corresponding to the foreground componentimage in the uncovered background region stored in the coefficient setmemory 4308. The mapping unit 4307-3 supplies the generated predictedimage to the synthesizing unit 4309.

[1197] The mapping unit 4307-4 generates a predicted image correspondingto the background component image in the covered background regionstored in the covered background region background component image framememory 4304 by the class classification adaptation processing based uponthe coefficient set corresponding to the background component image inthe covered background region stored in the coefficient set memory 4308.The mapping unit 4307-4 supplies the generated predicted image to thesynthesizing unit 4309.

[1198] The mapping unit 4307-5 generates a predicted image correspondingto the foreground component image in the covered background regionstored in the covered background region foreground component image framememory 4305 by the class classification adaptation processing based uponthe coefficient set corresponding to the foreground component image inthe covered background region stored in the coefficient set memory 4308.The mapping unit 4307-5 supplies the generated predicted image to thesynthesizing unit 4309.

[1199] The mapping unit 4307-6 generates a predicted image correspondingto the image in the foreground region stored in the foreground regionframe memory 4306 by the class classification adaptation processingbased upon the coefficient set corresponding to the foreground regionstored in the coefficient set memory 4308. The mapping unit 4307-6supplies the generated predicted image to the synthesizing unit 4309.

[1200] The synthesizing unit 4309 synthesizes the predicted imagecorresponding to the image in the background region supplied from themapping unit 4307-1, the predicted image corresponding to the backgroundcomponent image in the uncovered background region supplied from themapping unit 4307-2, the predicted image corresponding to the foregroundcomponent image in the uncovered background region supplied from themapping unit 4307-3, the predicted image corresponding to the backgroundcomponent image in the covered background region supplied from themapping unit 4307-4, the predicted image corresponding to the foregroundcomponent image in the covered background region supplied from themapping unit 4307-5, and the predicted image corresponding to the imagein the foreground region supplied from the mapping unit 4307-6, andsupplies the synthesized predicted image to frame memory 4310.

[1201] The frame memory 4310 stores the predicted image supplied fromthe synthesizing unit 4309, and also outputs the stored image as anoutput image.

[1202] Note that the mapping units 4307-1 through 4307-6 have the sameconfiguration as the mapping unit 1103, so description thereof will beomitted.

[1203] Referring to the images shown in FIG. 139A through FIG. 144B,description will be made with regard to the results of the processing ofthe image processing device according to the present invention havingthe separated image processing unit 4002 of which configuration is shownin FIG. 138.

[1204] In the processing for generating results shown by way ofexamples, the sum of the number of classes in the class classificationadaptation processing in the image processing device of the presentinvention is approximately the same as the number of classes in theconventional class classification adaptation processing. That is to say,the number of classes in the conventional class classificationadaptation processing is 2048, and the number of the classes in theclass classification adaptation processing in the image processingdevice of the present invention corresponding to the images in eachregion is arranged to be 512.

[1205] Also, the number of the prediction taps in the conventional classclassification adaptation processing and the number of the predictiontaps in the class classification adaptation processing for each regionin the image processing device of the present invention, are 9, i.e.,the same.

[1206] Referring to FIG. 139A through FIG. 141B, the results of theprediction in the covered background region will be described.

[1207]FIG. 139A is a diagram which illustrates an example of the imagein the mixed region of the tutor image. FIG. 139B is a diagram whichindicates the change in pixel value corresponding to the position in thespatial direction in the image in the mixed region of the tutor image.

[1208]FIG. 140A is a diagram which illustrates an example of the imagein the mixed region generated by the conventional class classificationadaptation processing corresponding to the tutor image illustrated inFIG. 139A. FIG. 140B is a diagram which indicates the change in thepixel value corresponding to the position in the spatial direction inthe image in the mixed region, generated by the conventional classclassification adaptation processing, corresponding to the tutor imageillustrated in FIG. 139A.

[1209]FIG. 141A is a diagram which illustrates an example of the imagein the mixed region, generated by the separated image processing unit4002 of which configuration is shown in FIG. 138, corresponding to thetutor image shown in FIG. 139A. FIG. 141B is a diagram which indicatesthe change in pixel value corresponding to the position in the spatialdirection in the image in the mixed region, generated by the separatedimage processing unit 4002 of which configuration is shown in FIG. 138,corresponding to the tutor image shown in FIG. 139A.

[1210] The pixel values in the image in the mixed region, generated bythe conventional class classification adaptation processing, change in astepped manner, as compared with the tutor image, and also are visuallyconfirmed to change in a stepped manner in the actual generated image.

[1211] Conversely, the pixel values in the image in the mixed region,generated by the separated image processing unit 4002 of whichconfiguration is shown in FIG. 138 change more smoothly as compared withconventional arrangement, and indicates change even closer to the tutorimage. Also, in the event of visually confirming the image generated bythe separated image processing unit 4002, the image is confirmed to bean even smoother image as compared with conventional arrangement.

[1212] The image in the mixed region, generated by the separated imageprocessing unit 4002 of which configuration is shown in FIG. 138,changes more smoothly as compared with the image generated by the inputimage being divided into the foreground region, mixed region, orbackground region.

[1213] Referring to FIG. 142A through FIG. 144B, description will bemade with regard to the results of the prediction in the foregroundregion wherein the pixel values change generally linearly with regard tothe pixel position.

[1214]FIG. 142A is a diagram which illustrates an example of the imagein the foreground region in the tutor image wherein the pixel valueschange generally linearly. FIG. 142B is a diagram which indicates changein the pixel value corresponding to the position in the spatialdirection in the image in the foreground region of the tutor imagewherein the pixel values change generally linearly.

[1215]FIG. 143A is a diagram which illustrates an example of the imagein the foreground region, corresponding to the image shown in FIG. 142A,generated by the conventional class classification adaptationprocessing. FIG. 143B is a diagram which indicates the change of thepixel values corresponding to the position in the spatial direction, inthe image in the foreground region, corresponding to the image shown inFIG. 142A, generated by the conventional class classification adaptationprocessing.

[1216]FIG. 144A is a diagram which illustrates an example of the imagein the foreground region corresponding to the image shown in FIG. 142A,generated by the separated image processing unit 4002 of whichconfiguration is shown in FIG. 138. FIG. 144B is a diagram whichindicates the change of the pixel values, corresponding to the positionin the spatial direction, in the image in the foreground region,corresponding to the image shown in FIG. 142A, generated by theseparated image processing unit 4002 of which configuration is shown inFIG. 138.

[1217] The pixel values in the image in the foreground region generatedby the conventional class classification adaptation processing change ina stepped manner as compared with the tutor image in the same manner asthe mixed region, and the change in a stepped manner can be visuallyrecognized in the actual image.

[1218] Conversely, the pixel values in the image in the foregroundregion generated by the separated image processing unit 4002 of whichconfiguration is shown in FIG. 138 change more smoothly as compared withconventional arrangement, and are extremely close to the values in thetutor image. In the visual confirmation of the image generated by theseparated image processing unit 4002, the difference between the imageand the tutor image could not be observed.

[1219]FIG. 145 is a flowchart which describes the processing for theimage by the image processing device of which configuration is shown inFIG. 133.

[1220] In Step S4001, the region specifying unit 103 specifies theforeground region, background region, covered background region, anduncovered background region, in the input image based upon the movementvector and the position information thereof supplied from the movementdetecting unit 102 and the input image. The processing in Step S4001 isthe same as the processing in Step S101, so description thereof will beomitted.

[1221] In Step S4002, the mixture ratio calculation unit 104 calculatesthe mixture ratio α based upon the region information supplied from theregion specifying unit 103, and the input image. The processing in StepS4002 is the same as the processing in Step S102, so description thereofwill be omitted.

[1222] In Step S4003, the foreground/background separation unit 4001separates the input image into the image in the foreground region, theimage in the background region, the foreground component image in thecovered background region, the background component image in the coveredbackground region, the foreground component image in the uncoveredbackground region, and the background component image in the uncoveredbackground region, based upon the region information supplied from theregion specifying unit 103 and the mixture ratio α supplied from themixture ratio calculation unit 104. Details of the processing for theseparation of the image by the foreground/background separation unit4001 will be described later.

[1223] In Step S4004, the separated image processing unit 4002 performsimage processing for each of the separated images, i.e., the image inthe foreground region, the image in the background region, theforeground component image in the covered background region, thebackground component image in the covered background region, theforeground component image in the uncovered background region, and thebackground component image in the uncovered background region, and theprocessing ends. Details of the image processing performed by theseparated image processing unit 4002 will be described later.

[1224] As described above, the image processing device according to thepresent invention separates the input image into the image in theforeground region, the image in the background region, the foregroundcomponent image in the covered background region, the backgroundcomponent image in the covered background region, the foregroundcomponent image in the uncovered background region, and the backgroundcomponent image in the uncovered background region, and performs theimage processing for each of the separated images, i.e., the image inthe foreground region, the image in the background region, theforeground component image in the covered background region, thebackground component image in the covered background region, theforeground component image in the uncovered background region, and thebackground component image in the uncovered background region.

[1225] The separation processing of the foreground and the background bythe foreground/background separation unit 4001 will now be described,with reference to the flowchart shown in FIG. 146. In Step S4101, theframe memory 4121 of the separation unit 4101 obtains the input image,and stores the frame #n which is the object of separation of theforeground and the background, as well as the previous frame #n−1 andthe following frame #n+1.

[1226] In Step S4102, the separation processing block 4122 of theseparation unit 4101 obtains the region information supplied from theregion specifying unit 103. In Step S4103, the separation processingblock 4122 of the separation unit 4101 obtains the mixture ratio αsupplied from the mixture ratio calculation unit 104.

[1227] In Step S4104, the uncovered region processing unit 4131 extractsthe background components from the pixel value of the pixel belonging tothe uncovered background region supplied from the frame memory 4121based upon the region information and the mixture ratio α and outputsthis as a background component image of the uncovered background region.

[1228] In Step S4105, the uncovered region processing unit 4131 extractsthe foreground components from the pixel value of the pixel belonging tothe uncovered background region supplied from the frame memory 4121based upon the region information and the mixture ratio α and outputsthis as a foreground component image of the uncovered background region.

[1229] In Step S4106, the covered region processing unit 4132 extractsthe background components from the pixel value of the pixel belonging tothe covered background region supplied from the frame memory 4121 basedupon the region information and the mixture ratio α and outputs this asa background component image of the covered background region.

[1230] In Step S4107, the covered region processing unit 4132 extractsthe foreground components from the pixel value of the pixel belonging tothe covered background region supplied from the frame memory 4121 basedupon the region information and the mixture ratio α and outputs this asa foreground component image of the covered background region, and theprocessing ends.

[1231] In this way, the foreground/background separation unit 4001 canseparate the foreground component and the background component from theinput image, based upon the region information and the mixture ratio α,and output a foreground component image consisting only of foregroundcomponents, and a background component image consisting only ofbackground components.

[1232] The learning processing for the separated image processing unit4002, of which the configuration is shown in FIG. 137, to generatecoefficient sets used for predicting pixel values from the classclassification adaptation processing, will be described with referenceto the flowchart in FIG. 147.

[1233] In Step S4201, the weighting averaging units 4207-1 through4207-6 generate student images of the background region image, theforeground region image, the background component image of the uncoveredbackground region, the foreground component image of the uncoveredbackground region, the background component image of the coveredbackground region, and the foreground component image of the coveredbackground region. That is to say, the weighting averaging unit 4207-1performs, for example, one-quarter weighting averaging of a backgroundregion image of a tutor image stored in background region tutor imageframe memory 4201, and generates a student image corresponding to thebackground region of the tutor image.

[1234] The weighting averaging unit 4207-2 performs, for example,one-quarter weighting averaging of a background component image in anuncovered background region image of a tutor image stored in uncoveredbackground region background component tutor image frame memory 4202,and generates a student image corresponding to the background componentimage in the uncovered background region of the tutor image.

[1235] The weighting averaging unit 4207-3 performs, for example,one-quarter weighting averaging of a foreground component image in anuncovered background region of a tutor image stored in uncoveredbackground region foreground component tutor image frame memory 4203,and generates a student image corresponding to the foreground componentimage in the uncovered background region of the tutor image.

[1236] The weighting averaging unit 4207-4 performs, for example,one-quarter weighting averaging of a background component image in acovered background region of a tutor image stored in covered backgroundregion background component tutor image frame memory 4204, and generatesa student image corresponding to the background component image in thecovered background region of the tutor image.

[1237] The weighting averaging unit 4207-5 performs, for example,one-quarter weighting averaging of a foreground component image in acovered background region of a tutor image stored in covered backgroundregion foreground component tutor image frame memory 4205, and generatesa student image corresponding to the foreground component image in thecovered background region of the tutor image.

[1238] The weighting averaging unit 4207-6 performs, for example,one-quarter weighting averaging of a foreground region image of a tutorimage stored in foreground region tutor image frame memory 4206, andgenerates a student image corresponding to the foreground region imageof the tutor image.

[1239] In Step S4202, the learning unit 4214-1 generates a coefficientset corresponding to the background region, based on the backgroundregion image of the tutor image stored in the background region tutorimage frame memory 4201, and the student image corresponding to thebackground region image of the tutor image stored in the backgroundregion student image frame memory 4208.

[1240] In Step S4203, the learning unit 4214-2 generates a coefficientset corresponding to the background component image of the uncoveredbackground region, based on the background component image of theuncovered background region of the tutor image which is stored inuncovered background region background component tutor image framememory 4202, and the student image, corresponding to the backgroundcomponent image of the uncovered background region of the tutor image,which is stored in uncovered background region background componentstudent image frame memory 4209.

[1241] In Step S4204, the learning unit 4214-3 generates a coefficientset corresponding to the foreground component image of the uncoveredbackground region, based on the foreground component image of theuncovered background region of the tutor image which is stored inuncovered background region foreground component tutor image framememory 4203, and the student image, corresponding to the foregroundcomponent image of the uncovered background region of the tutor image,which is stored in uncovered background region foreground componentstudent image frame memory 4210.

[1242] In Step S4205, the learning unit 4214-4 generates a coefficientset corresponding to the background component image of the coveredbackground region, based on the background component image of thecovered background region of the tutor image which is stored in coveredbackground region background component tutor image frame memory 4204,and the student image, corresponding to the background component imageof the covered background region of the tutor image, which is stored incovered background region background component student image framememory 4211.

[1243] In Step S4206, the learning unit 4214-5 generates a coefficientset corresponding to the foreground component image of the coveredbackground region, based on the foreground component image of thecovered background region of the tutor image which is stored in coveredbackground region foreground component tutor image frame memory 4205,and the student image, corresponding to the foreground component imageof the covered background region of the tutor image, which is stored incovered background region foreground component student image framememory 4212.

[1244] In Step S4207, the learning unit 4214-6 generates a coefficientset corresponding to the foreground region, based on the foregroundregion image of the tutor image stored in the foreground region tutorimage frame memory 4206, and the student image corresponding to theforeground region image of the tutor image stored in the foregroundregion student image frame memory 4213.

[1245] In Step S4208, the learning units 4214-1 through 4214-6respectively output a coefficient set corresponding to the backgroundregion, a coefficient set corresponding to the background componentimage of the uncovered background region, a coefficient setcorresponding to the foreground component image of the uncoveredbackground region, a coefficient set corresponding to the backgroundcomponent image of the covered background region, a coefficient setcorresponding to the foreground component image of the coveredbackground region, and a coefficient set corresponding to the foregroundregion, to coefficient set memory 4215. The coefficient set memory 4215stores the coefficient sets corresponding to the background region, theforeground region, the background component image of the uncoveredbackground region, the foreground component image of the uncoveredbackground region, the background component image of the coveredbackground region, and the foreground component image of the coveredbackground region, respectively, and the processing ends.

[1246] Thus, the separated image processing unit 4002 of whichconfiguration is shown in FIG. 137 can generate a coefficient setcorresponding to the background region, a coefficient set correspondingto the background component image of the uncovered background region, acoefficient set corresponding to the foreground component image of theuncovered background region, a coefficient set corresponding to thebackground component image of the covered background region, acoefficient set corresponding to the foreground component image of thecovered background region, and a coefficient set corresponding to theforeground region image.

[1247] The details of the processing in Step S4202 through Step S4207are the same as the processing described with reference to the flowchartin FIG. 103, so description thereof will be omitted.

[1248] Further, it is needless to say that the processing in Step S4202through Step S4207 can be performed in a serial manner or in a parallelmanner.

[1249] Next, the processing for the separated image processing unit 4002of which the configuration is shown in FIG. 138 to create images will bedescribed with reference to the flowchart in FIG. 148.

[1250] In Step S4301, the mapping unit 4307-1 predicts an imagecorresponding to the background region image stored in the backgroundregion frame memory 4301 by class classification adaptation processing,based on the coefficient set corresponding to the background regionwhich is stored in the coefficient set memory 4308.

[1251] In Step S4302, the mapping unit 4307-2 predicts an imagecorresponding to the background component image of the uncoveredbackground region which is stored in uncovered background regionbackground component image frame memory 4302 by class classificationadaptation processing, based on the coefficient set corresponding to thebackground component image of the uncovered background region which isstored in coefficient set memory 4308.

[1252] In Step S4303, the mapping unit 4307-3 predicts an imagecorresponding to the foreground component image of the uncoveredbackground region which is stored in uncovered background regionforeground component image frame memory 4303 by class classificationadaptation processing, based on the coefficient set corresponding to theforeground component image of the uncovered background region which isstored in coefficient set memory 4308.

[1253] In Step S4304, the mapping unit 4307-4 predicts an imagecorresponding to the background component image of the coveredbackground region which is stored in covered background regionbackground component image frame memory 4304 by class classificationadaptation processing, based on the coefficient set corresponding to thebackground component image of the covered background region which isstored in coefficient set memory 4308.

[1254] In Step S4305, the mapping unit 4307-5 predicts an imagecorresponding to the foreground component image of the coveredbackground region which is stored in covered background regionforeground component image frame memory 4305 by class classificationadaptation processing, based on the coefficient set corresponding to theforeground component image of the covered background region which isstored in coefficient set memory 4308.

[1255] In Step S4306, the mapping unit 4307-6 predicts an imagecorresponding to the foreground region image stored in the foregroundregion frame memory 4306 by class classification adaptation processing,based on the coefficient set corresponding to the foreground regionwhich is stored in the coefficient set memory 4308.

[1256] In Step S4307, the synthesizing unit 4309 synthesizes thepredicted image corresponding to the background region image, thepredicted image corresponding to the background component image of theuncovered background region, the predicted image corresponding to theforeground component image of the uncovered background region, thepredicted image corresponding to the background component image of thecovered background region, the predicted image corresponding to theforeground component image of the covered background region, and thepredicted image corresponding to the foreground region. The synthesizingunit 4309 supplies the synthesized image to the frame memory 4310. Theframe memory 4310 stores the synthesized image supplied from thesynthesizing unit 4309.

[1257] In Step S4308, the frame memory 4310 outputs the synthesizedimage stored therein, and the processing ends.

[1258] Thus, an image processing device having the separated imageprocessing unit 4002 of which configuration is shown in FIG. 138 cangenerate predicted images for each of the separated background regionimage, background component image of the uncovered background region,foreground component image of the uncovered background region,background component image of the covered background region, foregroundcomponent image of the covered background region, and foreground regionimage.

[1259] The details of the processing in Step S4301 through Step S4306are the same as the processing described with reference to the flowchartin FIG. 109, so description thereof will be omitted.

[1260] Further, it is needless to say that the processing in Step S4301through Step S4306 can be performed in a serial manner or in a parallelmanner.

[1261]FIG. 149 is a block diagram illustrating the configuration of aseparated image processing unit 4002 which applies edge enhancementprocessing with different effects to each of background region image,background component image of the uncovered background region,foreground component image of the uncovered background region,background component image of the covered background region, foregroundcomponent image of the covered background region, and foreground regionimage.

[1262] The background region frame memory 4501 stores the backgroundregion image made up of pixels belonging to the background region,supplied from the foreground/background separation unit 4001. Thebackground region frame memory 4501 supplies the stored backgroundregion image to the edge enhancing unit 4507-1.

[1263] The uncovered background region background component image framememory 4502 stores the background component image of the uncoveredbackground region, supplied from the foreground/background separationunit 4001. The uncovered background region background component imageframe memory 4502 supplies the stored background component image of theuncovered background region to the edge enhancing unit 4507-2.

[1264] The uncovered background region foreground component image framememory 4503 stores the foreground component image of the uncoveredbackground region, supplied from the foreground/background separationunit 4001. The uncovered background region foreground component imageframe memory 4503 supplies the stored foreground component image of theuncovered background region to the edge enhancing unit 4507-3.

[1265] The covered background region background component image framememory 4504 stores the background component image of the coveredbackground region, supplied from the foreground/background separationunit 4001. The covered background region background component imageframe memory 4504 supplies the stored background component image of thecovered background region to the edge enhancing unit 4507-4.

[1266] The covered background region foreground component image framememory 4505 stores the foreground component image of the coveredbackground region, supplied from the foreground/background separationunit 4001. The covered background region foreground component imageframe memory 4505 supplies the stored foreground component image of thecovered background region to the edge enhancing unit 4507-5.

[1267] The foreground region frame memory 4506 stores the foregroundregion image made up of pixels belonging to the foreground region,supplied from the foreground/background separation unit 4001. Theforeground region frame memory 4506 supplies the stored foregroundregion image to the edge enhancing unit 4507-6.

[1268] The edge enhancing unit 4507-1 applies edge enhancementprocessing suitable to the image of the background region, to thebackground region image stored in the background region frame memory4501, and supplies the background region image which has been subjectedto edge enhancement to the synthesizing unit 4508.

[1269] For example, the edge enhancing unit 4507-1 performs theprocessing of edge enhancement, which further enhances the edge for thebackground region image which is a still image, as compared with theforeground region. Thus, the sense-of-resolution of the backgroundregion image can be improved without unnatural degradation of the imageoccurring in the event of applying the processing of edge enhancement toa moving image.

[1270] The edge enhancing unit 4507-2 applies edge enhancementprocessing suitable to the background component image of the uncoveredbackground region, to the image stored in the uncovered backgroundregion background component image frame memory 4502, and supplies theimage which has been subjected to edge enhancement to the synthesizingunit 4508.

[1271] For example, the edge enhancing unit 4507-2 performs theprocessing of edge enhancement, which further enhances the edge for thebackground component image of the uncovered background region which is astill image, as compared with the foreground region. Thus, thesense-of-resolution of the background region image can be improvedwithout unnatural degradation of the image occurring in the event ofapplying the processing of edge enhancement to a moving image.

[1272] The edge enhancing unit 4507-3 applies edge enhancementprocessing suitable to the foreground component image of the uncoveredbackground region, to the image stored in the uncovered backgroundregion foreground component image frame memory 4503, and supplies theimage which has been subjected to edge enhancement to the synthesizingunit 4508.

[1273] For example, the edge enhancing unit 4507-3 performs theprocessing of edge enhancement, which enhances the edge for theforeground component image of the covered background region made up ofmoving foreground components less than as compared with the backgroundregion. Thus, the sense-of-resolution of the foreground component imageof the covered background region can be improved without unnaturaldegradation of the image occurring in the event of applying theprocessing of edge enhancement to a moving image.

[1274] The edge enhancing unit 4507-4 applies edge enhancementprocessing suitable to the background component image of the coveredbackground region, to the image stored in the covered background regionbackground component image frame memory 4504, and supplies the imagewhich has been subjected to edge enhancement to the synthesizing unit4508.

[1275] For example, the edge enhancing unit 4507-4 performs theprocessing of edge enhancement, which further enhances the edge for thebackground component image of the covered background region which is astill image, as compared with the foreground region. Thus, thesense-of-resolution of the background region image can be improvedwithout unnatural degradation of the image occurring in the event ofapplying the processing of edge enhancement to a moving image.

[1276] The edge enhancing unit 4507-5 applies edge enhancementprocessing suitable to the foreground component image of the coveredbackground region, to the image stored in the covered background regionforeground component image frame memory 4505, and supplies the imagewhich has been subjected to edge enhancement to the synthesizing unit4508.

[1277] For example, the edge enhancing unit 4507-5 performs theprocessing of edge enhancement, which enhances the edge for theforeground component image of the covered background region made up ofmoving foreground components less than as compared with the backgroundregion. Thus, the sense-of-resolution of the foreground region image ofthe covered background region can be improved without unnaturaldegradation of the image occurring in the event of applying theprocessing of edge enhancement to a moving image.

[1278] The edge enhancing unit 4507-6 applies edge enhancementprocessing suitable to the foreground region image, to the foregroundregion image stored in the foreground region frame memory 4506, andsupplies the foreground region image which has been subjected to edgeenhancement to the synthesizing unit 4508.

[1279] For example, the edge enhancing unit 4507-6 performs theprocessing of edge enhancement, which enhances the edge of the movingforeground region image less than as compared with the backgroundregion. Thus, the sense-of-resolution of the foreground region image canbe improved without unnatural degradation of the image occurring in theevent of applying the processing of edge enhancement to a moving image.

[1280] The synthesizing unit 4508 synthesizes the background regionimage subjected to edge enhancing that has been supplied from the edgeenhancing unit 4507-1, the background component image of the uncoveredbackground region subjected to edge enhancing that has been suppliedfrom the edge enhancing unit 4507-2, the foreground component image ofthe uncovered background region subjected to edge enhancing that hasbeen supplied from the edge enhancing unit 4507-3, the backgroundcomponent image of the covered background region subjected to edgeenhancing that has been supplied from the edge enhancing unit 4507-4,the foreground component image of the covered background regionsubjected to edge enhancing that has been supplied from the edgeenhancing unit 4507-5, and the foreground region image subjected to edgeenhancing that has been supplied from the edge enhancing unit 4507-6,and supplies the synthesized image to the frame memory 4509.

[1281] The frame memory 4509 stores the synthesized image supplied fromthe synthesizing unit 4508, and also outputs the stored image as anoutput image.

[1282] Thus, the separated image processing unit 4002 of whichconfiguration is shown in FIG. 149 can apply edge enhancement processingcorresponding to the image nature of each of the background regionimage, background component image of the uncovered background region,foreground component image of the uncovered background region,background component image of the covered background region, foregroundcomponent image of the covered background region, and foreground regionimage, so the sense-of-resolution of the foreground region image can beimproved without unnatural degradation of the image occurring.

[1283] The edge enhancing units 4507-1 through 4507-6 have the sameconfiguration as the edge enhancing unit 1203, so description thereofwill be omitted.

[1284]FIG. 150 is a diagram explaining the processing of the separatedimage processing unit 4002 of which the configuration is shown in FIG.149. The region specifying unit 103 specifies the foreground region,background region, covered background region, and uncovered backgroundregion, of the input image.

[1285] The input image regarding which the regions have been specifiedand the mixture ratio α has been detected by the mixture ratiocalculation unit 104 is separated into a foreground region image,background region image, foreground component image of the coveredbackground region, background component image of the covered backgroundregion, foreground component image of the uncovered background region,and background component image of the uncovered background region, bythe foreground/background separation unit 4001.

[1286] The separated image processing unit 4002 applies edge enhancementprocessing to each of the separated foreground region image, backgroundregion image, foreground component image of the covered backgroundregion, background component image of the covered background region,foreground component image of the uncovered background region, andbackground component image of the uncovered background region.

[1287] The background region image, foreground component image of thecovered background region, background component image of the coveredbackground region, foreground component image of the uncoveredbackground region, and background component image of the uncoveredbackground region, which have each been subjected to edge enhancement,are synthesized into one image.

[1288]FIG. 151 is a flowchart explaining the processing of images withthe image processing device of which the configuration is indicated inFIG. 133.

[1289] In Step S4501, the region specifying unit 103 specifies theforeground region, background region, covered background region, anduncovered background region, of the input image, based on the movementvector and the position information thereof supplied from the movementdetecting unit 102. The processing in Step S4501 is the same as theprocessing in Step S4001, so description thereof will be omitted.

[1290] In Step S4502, the mixture ratio calculation unit 104 calculatesthe mixture ratio α based on the region information supplied from theregion specifying unit 103 and the input image. The processing in StepS4502 is the same as the processing in Step S4002, so descriptionthereof will be omitted.

[1291] In Step S4503, the foreground/background separation unit 4001separates the input image into a foreground region image, backgroundregion image, foreground component image of the covered backgroundregion, background component image of the covered background region,foreground component image of the uncovered background region, andbackground component image of the uncovered background region, based onthe region information supplied from the region specifying unit 103 andthe mixture ratio α supplied from the mixture ratio calculation unit104. The processing in Step S4503 is the same as the processing in StepS4003, so description thereof will be omitted.

[1292] In Step S4504, the separated image processing unit 4002 appliesedge enhancement processing to each of the separated foreground regionimage, background region image, foreground component image of thecovered background region, background component image of the coveredbackground region, foreground component image of the uncoveredbackground region, and background component image of the uncoveredbackground region, and the processing ends. Details of the imageprocessing which the separated image processing unit 4002 executes willbe described later.

[1293] Thus, the image processing device according to the presentinvention separates an input image into a foreground region image,background region image, foreground component image of the coveredbackground region, background component image of the covered backgroundregion, foreground component image of the uncovered background region,and background component image of the uncovered background region, andexecutes edge enhancement processing for each separated foregroundregion image, background region image, foreground component image of thecovered background region, background component image of the coveredbackground region, foreground component image of the uncoveredbackground region, and background component image of the uncoveredbackground region.

[1294] Next, the edge enhancement processing of the separated imageprocessing unit 4002, corresponding to Step S4504, will be describedwith reference to the flowchart in FIG. 152.

[1295] In Step S4521, the edge enhancing unit 4507-1 performs edgeenhancement of the background region image stored in the backgroundregion frame memory 4501, by edge enhancement processing correspondingto the nature of the background region image.

[1296] In Step S4522, the edge enhancing unit 4507-2 performs edgeenhancement of the background component image of the uncoveredbackground region, stored in the uncovered background region backgroundcomponent image frame memory 4502, by edge enhancement processingcorresponding to the nature of the background component image of theuncovered background region.

[1297] In Step S4523, the edge enhancing unit 4507-3 performs edgeenhancement of the foreground component image of the uncoveredbackground region, stored in the uncovered background region foregroundcomponent image frame memory 4503, by edge enhancement processingcorresponding to the nature of the foreground component image of theuncovered background region.

[1298] In Step S4524, the edge enhancing unit 4507-4 performs edgeenhancement of the background component image of the covered backgroundregion, stored in the covered background region background componentimage frame memory 4504, by edge enhancement processing corresponding tothe nature of the background component image of the covered backgroundregion.

[1299] In Step S4525, the edge enhancing unit 4507-5 performs edgeenhancement of the foreground component image of the covered backgroundregion, stored in the covered background region foreground componentimage frame memory 4505, by edge enhancement processing corresponding tothe nature of the foreground component image of the covered backgroundregion.

[1300] In Step S4526, the edge enhancing unit 4507-6 performs edgeenhancement of the foreground region image stored in the foregroundregion frame memory 4506, by edge enhancement processing correspondingto the nature of the foreground region image.

[1301] In Step S4527, the synthesizing unit 4508 synthesizes theforeground region image, background region image, foreground componentimage of the covered background region, background component image ofthe covered background region, foreground component image of theuncovered background region, and background component image of theuncovered background region, regarding which each has been subjected toedge enhancement. The synthesizing unit 4508 supplies the synthesizedimage to the frame memory 4509. The frame memory 4509 stores the imagesupplied from the synthesizing unit 4508.

[1302] In Step S4528, the frame memory 4509 outputs the synthesizedimage stored therein, and the processing ends.

[1303] Thus, the separated image processing unit 4002 of which theconfiguration shown in FIG. 149 can execute edge enhancement processingcorresponding to the nature of each of the foreground region image,background region image, foreground component image of the coveredbackground region, background component image of the covered backgroundregion, foreground component image of the uncovered background region,and background component image of the uncovered background region, sothe sense-of-resolution can be improved without causing unnaturaldistortion in moving images.

[1304] Note that it is needless to say that the processing in Step S4521through Step S4526 can be performed in a serial manner or in a parallelmanner.

[1305]FIG. 153 is a block diagram illustrating another configuration ofthe functions of the image processing device for separating an inputimage and processing each separated image. While the image processingdevice shown in FIG. 133 performs region specification and calculationof the mixture ratio α serially, the image processing device shown inFIG. 153 performs region specification and calculation of the mixtureratio α in parallel.

[1306] Portions which are the same as the function in the block diagramshown in FIG. 133 are denoted with the same numerals, and descriptionthereof will be omitted.

[1307] The input image is supplied to the object extracting unit 101,region specifying unit 103, mixture ratio calculation unit 3001, andforeground/background separation unit 4601.

[1308] Based on an input image, the mixture ratio calculation unit 3001calculates an estimated mixture ratio in a case wherein a pixel isassumed to belong to the covered background region, and an estimatedmixture ratio in a case wherein the pixel is assumed to belong to theuncovered background region, for each of the pixels contained in theinput image, and supplies the estimated mixture ratio in a case whereinthe pixel is assumed to belong to the covered background region and theestimated mixture ratio in a case wherein the pixel is assumed to belongto the uncovered background region, thus calculated, to theforeground/background separation unit 4601.

[1309] Based on the estimated mixture ratio in a case wherein the pixelis assumed to belong to the covered background region and the estimatedmixture ratio in a case wherein the pixel is assumed to belong to theuncovered background region, supplied from the mixture ratio calculationunit 3001, and the region information supplied from the regionspecifying unit 103, the foreground/background separation unit 4601separates the input image into a foreground region image, backgroundregion image, foreground component image of the covered backgroundregion, background component image of the covered background region,foreground component image of the uncovered background region, andbackground component image of the uncovered background region, andsupplies the separated image to the separated image processing unit4002.

[1310]FIG. 154 is a block diagram illustrating an example of theconfiguration of the foreground/background separation unit 4601.

[1311] Portions which are the same as the foreground/backgroundseparation unit 4001 shown in FIG. 135 are denoted with the samenumeral, and description thereof will be omitted.

[1312] A selecting unit 4621 selects one or the other of the estimatedmixture ratio in a case wherein the pixel is assumed to belong to thecovered background region and the estimated mixture ratio in a casewherein the pixel is assumed to belong to the uncovered backgroundregion, supplied from the mixture ratio calculation unit 3001, based onthe region information supplied from the region specifying unit 103, andsupplies the selected estimated mixture ratio to the separation unit4101 as mixture ratio α.

[1313] The separation unit 4101 extracts the foreground component andbackground component from the pixel values of the pixels belonging tothe mixed region, based on the mixture ratio α supplied from theselecting unit 4621 and the region information, and separates into abackground component image of the uncovered background region, aforeground component image of the uncovered background region, abackground component image of the covered background region, and aforeground component image of the covered background region.

[1314] The separation unit 4101 may have the same configuration as theconfiguration shown in FIG. 136.

[1315] Thus, the image processing device of which the configuration isshown in FIG. 153 is capable of executing processing for each of theforeground region image, background region image, foreground componentimage of the covered background region, background component image ofthe covered background region, foreground component image of theuncovered background region, and background component image of theuncovered background region, corresponding to the respective nature ofeach.

[1316]FIG. 155 is a block diagram illustrating yet another configurationof the functions of the image processing device.

[1317] Portions which are the same as those shown in FIG. 11 are denotedwith the same numerals, and description thereof will be omitted.

[1318] The input image supplied to the image processing device issupplied to the object extracting unit 101, region specifying unit 103,and region processing unit 5001.

[1319] The object extracting unit 101 roughly extracts the image objectcorresponding to the foreground object contained in the input image, andsupplies the extracted image object to the movement detecting unit 102.The object extracting unit 101 roughly extracts the image objectcorresponding to the background object contained in the input image, andsupplies the extracted image object to the movement detecting unit 102.

[1320] The movement detecting unit 102 calculates the movement vectorsof the image object corresponding to the roughly extracted foregroundobjects by techniques such as block matching, gradation, phasecorrelation, and pixel recursion, or the like, and provides thecalculated movement vectors and movement vector position information tothe region specifying unit 103.

[1321] The region specifying unit 103 specifies each of the pixels ofthe image input, into foreground region, background region, or mixedregion, and supplies region information indicating which of theforeground region, background region, or mixed region, that each pixelbelongs to, to the region processing unit 5001.

[1322] The region processing unit 5001 divides the input image into eachof the foreground region, background region, and mixed region, based onthe region information supplied from the region specifying unit 103, andexecutes image processing for each of the divided input images. Forexample, the region processing unit 5001 divides the input image intoeach of the foreground region, background region, and mixed region, andgenerates coefficients used for class classification adaptationprocessing for generating images with higher resolution, for each of thedivided input images.

[1323] For example, the region processing unit 5001 divides the inputimage into each of the foreground region, background region, and mixedregion, and creates images with higher resolution by applying classclassification adaptation processing to each of the divided inputimages.

[1324]FIG. 156 is a diagram describing the processing of the regionprocessing unit 5001 of which the configuration is shown in FIG. 155.The region processing unit 5001 individually calculates coefficient setscorresponding to the background region, coefficient sets correspondingto the uncovered background region, coefficient sets corresponding tothe foreground region, and coefficient sets corresponding to the coveredbackground region.

[1325] The coefficient sets corresponding to the background region areused for predicting pixel values in the background region, in the classclassification adaptation processing for predicting pixel values. Thecoefficient sets corresponding to the uncovered background region areused for predicting pixel values in the uncovered background region, inthe class classification adaptation processing for predicting pixelvalues.

[1326] The coefficient sets corresponding to the covered backgroundregion are used for predicting pixel values in the covered backgroundregion, in the class classification adaptation processing for predictingpixel values. The coefficient sets corresponding to the foregroundregion are used for predicting pixel values in the foreground region, inthe class classification adaptation processing for predicting pixelvalues.

[1327] The predicted image corresponding to the background image, thepredicted image corresponding to the uncovered background region, thepredicted image corresponding to the covered background region, and thepredicted image corresponding to the foreground image, are synthesizedinto one predicted image.

[1328]FIG. 157 is a block diagram illustrating the configuration of theregion processing unit 5001 which generates coefficient sets used forclass classification adaptation processing for generating images withhigher resolution in the spatial direction. Tutor image frame memory5101 stores input images which are HD images, for example, in incrementsof frames. The tutor image frame memory 5101 supplies the stored inputimage to a region dividing unit 5102.

[1329] The region dividing unit 5102 divides the tutor image into thebackground region, foreground region, covered background region, oruncovered background region, based on the region information suppliedfrom the region specifying unit 103.

[1330] The region dividing unit 5102 supplies images made up of pixelsbelonging to the background region of the tutor image, which is thedivided tutor image, to background region tutor image frame memory 5103,supplies images made up of pixels belonging to the uncovered backgroundregion of the tutor image to uncovered background region tutor imageframe memory 5104, supplies images made up of pixels belonging to thecovered background region of the tutor image to covered backgroundregion tutor image frame memory 5105, and supplies images made up ofpixels belonging to the foreground region of the tutor image toforeground region tutor image frame memory 5106.

[1331] The background region tutor image frame memory 5103 stores imagesmade up of pixels belonging to the background region of the tutor image,supplied from the region dividing unit 5102. The background region tutorimage frame memory 5103 supplies images made up of pixels belonging tothe background region of the tutor image stored therein to a weightedaveraging unit 5107-1 and learning unit 5112-1.

[1332] The uncovered background region tutor image frame memory 5104stores images made up of pixels belonging to the uncovered backgroundregion of the tutor image, supplied from the region dividing unit 5102.The uncovered background region tutor image frame memory 5104 suppliesimages made up of pixels belonging to the uncovered background region ofthe tutor image stored therein to a weighted averaging unit 5107-2 andlearning unit 5112-2.

[1333] The covered background region tutor image frame memory 5105stores images made up of pixels belonging to the covered backgroundregion of the tutor image, supplied from the region dividing unit 5102.The covered background region tutor image frame memory 5105 suppliesimages made up of pixels belonging to the covered background region ofthe tutor image stored therein to a weighted averaging unit 5107-3 andlearning unit 5112-3.

[1334] The foreground region tutor image frame memory 5106 stores imagesmade up of pixels belonging to the foreground region of the tutor image,supplied from the region dividing unit 5102. The foreground region tutorimage frame memory 5106 supplies images made up of pixels belonging tothe foreground region of the tutor image stored therein to a weightedaveraging unit 5107-4 and learning unit 5112-4.

[1335] The weighted averaging unit 5107-1 performs one-quarter weightedaveraging on the image made up of pixels belonging to the backgroundregion of the tutor image which is an HD image, for example, suppliedfrom the background region tutor image frame memory 5103, so as togenerate an SD image which is a student image, and supplies thegenerated SD image to background region student image frame memory 5108.

[1336] The background region student image frame memory 5108 storesstudent images corresponding to the image made up of pixels belonging tothe background region of the tutor image supplied from the weightedaveraging unit 5107-1. The background region student image frame memory5108 supplies the student image corresponding to the image made up ofpixels belonging to the background region of the tutor image storedtherein, to the learning unit 5112-1.

[1337] The weighted averaging unit 5107-2 performs one-quarter weightedaveraging, for example, on the image made up of pixels belonging to theuncovered background region of the tutor image which is an HD image,supplied from the uncovered background region tutor image frame memory5104, so as to generate an SD image which is a student image, andsupplies the generated SD image to uncovered background region studentimage frame memory 5109.

[1338] The uncovered background region student image frame memory 5109stores student images which are SD images corresponding to the imagemade up of pixels belonging to the uncovered background region of thetutor image supplied from the weighted averaging unit 5107-2. Theuncovered background region student image frame memory 5109 supplies thestudent image corresponding to the image made up of pixels belonging tothe uncovered background region of the tutor image stored therein, tothe learning unit 5112-2.

[1339] The weighted averaging unit 5107-3 performs one-quarter weightedaveraging, for example, on the image made up of pixels belonging to thecovered background region of the tutor image, supplied from the coveredbackground region tutor image frame memory 5105, so as to generate an SDimage which is a student image, and supplies the generated SD image tocovered background region student image frame memory 5110.

[1340] The covered background region student image frame memory 5110stores student images which are SD images corresponding to the imagemade up of pixels belonging to the covered background region of thetutor image supplied from the weighted averaging unit 5107-3. Thecovered background region student image frame memory 5110 supplies thestudent image corresponding to the image made up of pixels belonging tothe covered background region of the tutor image stored therein, to thelearning unit 5112-3.

[1341] The weighted averaging unit 5107-4 performs one-quarter weightedaveraging on the image made up of pixels belonging to the foregroundregion of the tutor image which is an HD image, for example, suppliedfrom the foreground region tutor image frame memory 5106, so as togenerate an SD image which is a student image, and supplies thegenerated SD image to foreground region student image frame memory 5111.

[1342] The foreground region student image frame memory 5111 storesstudent images which are SD images corresponding to the image made up ofpixels belonging to the foreground region of the tutor image suppliedfrom the weighted averaging unit 5107-4. The foreground region studentimage frame memory 5111 supplies the student image corresponding to theimage made up of pixels belonging to the foreground region of the tutorimage stored therein, to the learning unit 5112-4.

[1343] The learning unit 5112-1 generates a coefficient setcorresponding to the background region, based on the image made up ofpixels belonging to the background region of the tutor image suppliedfrom the background region tutor image frame memory 5103 and the studentimage corresponding to the image made up of pixels belonging to thebackground region of the tutor image supplied from the background regionstudent image frame memory 5108, and supplies the generated coefficientset to coefficient set memory 5113.

[1344] The learning unit 5112-2 generates a coefficient setcorresponding to the uncovered background region, based on the imagemade up of pixels belonging to the uncovered background region of thetutor image supplied from the uncovered background region tutor imageframe memory 5104 and the student image corresponding to the image madeup of pixels belonging to the uncovered background region of the tutorimage supplied from the uncovered background region student image framememory 5109, and supplies the generated coefficient set to coefficientset memory 5113.

[1345] The learning unit 5112-3 generates a coefficient setcorresponding to the covered background region, based on the image madeup of pixels belonging to the covered background region of the tutorimage supplied from the covered background region tutor image framememory 5105 and the student image corresponding to the image made up ofpixels belonging to the covered background region of the tutor imagesupplied from the covered background region student image frame memory5110, and supplies the generated coefficient set to coefficient setmemory 5113.

[1346] The learning unit 5112-4 generates a coefficient setcorresponding to the foreground region, based on the image made up ofpixels belonging to the foreground region of the tutor image suppliedfrom the foreground region tutor image frame memory 5106 and the studentimage corresponding to the image made up of pixels belonging to theforeground region of the tutor image supplied from the foreground regionstudent image frame memory 5111, and supplies the generated coefficientset to coefficient set memory 5113.

[1347] The coefficient set memory 5113 stores the coefficient setcorresponding to the background region supplied from the learning unit5112-1, the coefficient set corresponding to the uncovered backgroundregion supplied from the learning unit 5112-2, the coefficient setcorresponding to the covered background region supplied from thelearning unit 5112-3, and the coefficient set corresponding to theforeground region supplied from the learning unit 5112-4.

[1348] The learning unit 5112-1 through learning unit 5112-4 have thesame configuration as the learning unit 1006, so description thereofwill be omitted.

[1349]FIG. 158 is a block diagram illustrating the configuration of aregion processing unit 5001 for executing class classificationadaptation processing, and generating images with higher resolution inthe spatial direction. The frame memory 5201 stores input images whichare SD images, for example, in increments of frames. The frame memory5201 supplies the stored input images to the region dividing unit 5202.

[1350] The region dividing unit 5202 divides the input image into thebackground region, foreground region, covered background region, oruncovered background region, based on the region information suppliedfrom the region specifying unit 103. That is to say, the region dividingunit 5202 takes the divided input image and supplies an image made up ofpixels belonging to the background region to the background region framememory 5203, supplies an image made up of pixels belonging to theuncovered background region to the uncovered background region framememory 5204, supplies an image made up of pixels belonging to thecovered background region to the covered background region frame memory5205, and supplies an image made up of pixels belonging to theforeground region to the foreground region frame memory 5206.

[1351] The background region frame memory 5203 stores the image made upof pixels belonging to the background region, supplied from the regiondividing unit 5202. The background region frame memory 5203 supplies theimage made up of pixels belonging to the background region storedtherein, to the mapping unit 5207-1.

[1352] The uncovered background region frame memory 5204 stores theimage made up of pixels belonging to the uncovered background region,supplied from the region dividing unit 5202. The uncovered backgroundregion frame memory 5204 supplies the image made up of pixels belongingto the uncovered background region stored therein, to the mapping unit5207-2.

[1353] The covered background region frame memory 5205 stores the imagemade up of pixels belonging to the covered background region, suppliedfrom the region dividing unit 5202. The covered background region framememory 5205 supplies the image made up of pixels belonging to thecovered background region stored therein, to the mapping unit 5207-3.

[1354] The foreground region frame memory 5206 stores the image made upof pixels belonging to the foreground region, supplied from the regiondividing unit 5202. The foreground region frame memory 5206 supplies theimage made up of pixels belonging to the foreground region storedtherein, to the mapping unit 5207-4.

[1355] The mapping unit 5207-1 generates a predicted image correspondingto the image made up of pixels belonging to the background region storedin the background region frame memory 5203 by class classificationadaptation processing, based on the coefficient set corresponding to thebackground region, stored in the coefficient set memory 5208. Themapping unit 5207-1 supplies the generated predicted image to asynthesizing unit 5209.

[1356] The mapping unit 5207-2 generates a predicted image correspondingto the image made up of pixels belonging to the uncovered backgroundregion stored in the uncovered background region frame memory 5204 byclass classification adaptation processing, based on the coefficient setcorresponding to the uncovered background region, stored in thecoefficient set memory 5208. The mapping unit 5207-2 supplies thegenerated predicted image to the synthesizing unit 5209.

[1357] The mapping unit 5207-3 generates a predicted image correspondingto the image made up of pixels belonging to the covered backgroundregion stored in the covered background region frame memory 5205 byclass classification adaptation processing, based on the coefficient setcorresponding to the covered background region, stored in thecoefficient set memory 5208. The mapping unit 5207-3 supplies thegenerated predicted image to the synthesizing unit 5209.

[1358] The mapping unit 5207-4 generates a predicted image correspondingto the image made up of pixels belonging to the foreground region storedin the foreground region frame memory 5206 by class classificationadaptation processing, based on the coefficient set corresponding to theforeground region, stored in the coefficient set memory 5208. Themapping unit 5207-4 supplies the generated predicted image to thesynthesizing unit 5209.

[1359] The synthesizing unit 5209 synthesizes the predicted imagecorresponding to the image made up of pixels belonging to the backgroundregion that has been supplied from the mapping unit 5207-1, thepredicted image corresponding to the image made up of pixels belongingto the uncovered background region that has been supplied from themapping unit 5207-2, the predicted image corresponding to the image madeup of pixels belonging to the covered background region that has beensupplied from the mapping unit 5207-3, and the predicted imagecorresponding to the image made up of pixels belonging to the foregroundregion that has been supplied from the mapping unit 5207-4, and suppliesthe synthesized predicted image to frame memory 5210.

[1360] The frame memory 5210 stores the predicted images supplied fromthe synthesizing unit 5209, and also outputs the stored images as outputimages.

[1361] The mapping unit 5207-1 through mapping unit 5207-4 have the sameconfiguration as the mapping unit 1103, and accordingly descriptionthereof will be omitted.

[1362] An example of the results of processing my the image processingdevice according to the present invention having the region processingunit 5001 of which configuration is shown in FIG. 158 will be describedwith reference to the images shown in FIG. 159A through FIG. 164B.

[1363] In the processing for generating the results shown in theexample, the sum of the number of classes in the class classificationadaptation processing of the image processing device according to thepresent invention is the same as the number of classes in conventionalclass classification adaptation processing. That is, the number ofclasses in conventional class classification adaptation processing is2048, and the number of classes in the class classification adaptationprocessing for each of the regions with the image processing deviceaccording to the present invention is 3112.

[1364] Also, the number of prediction taps in the conventional classclassification adaptation processing and the number of prediction tapsin the class classification adaptation processing for each of theregions with the image processing device according to the presentinvention is nine, which is the same.

[1365] The results of prediction for the covered background region willbe described with reference to FIG. 159A through FIG. 161B.

[1366]FIG. 159A is a diagram illustrating an example of an image in themixed region of a tutor image. FIG. 159B is a diagram illustrating thechange of pixel values corresponding to the position in the spatialdirection of the image in the mixed region of the tutor image.

[1367]FIG. 160A is a diagram illustrating an example of an image in themixed region, generated by conventional class classification adaptationprocessing, corresponding to the tutor image shown in FIG. 159A. FIG.160B is a diagram illustrating the change of pixel values correspondingto the position in the spatial direction of the image in the mixedregion, generated by conventional class classification adaptationprocessing, corresponding to the tutor image shown in FIG. 159A.

[1368]FIG. 161A is a diagram illustrating an example of an image in themixed region, generated by the region processing unit 5001 of which theconfiguration is shown in FIG. 158, corresponding to the tutor imageshown in FIG. 159A. FIG. 161B is a diagram illustrating the change ofpixel values corresponding to the spatial direction of the image in themixed region, generated by the region processing unit 5001 of which theconfiguration is shown in FIG. 158, corresponding to the tutor imageshown in FIG. 159A.

[1369] The pixel values in the image in the mixed region, generated bythe conventional class classification adaptation processing, change in astepped manner, as compared with the tutor image, and also are visuallyconfirmed to change in a stepped manner in the actually-generated image.

[1370] Conversely, the pixel values in the image in the mixed region,generated by the region processing unit 5001 of which configuration isshown in FIG. 158, change more smoothly as compared with conventionalarrangement, and indicates change even closer to the tutor image. Also,in the event of visually confirming the image generated by the regionprocessing unit 5001, the image is confirmed to be an even smootherimage as compared with the conventional arrangement.

[1371] Referring to FIG. 162A through FIG. 164B, description will bemade with regard to the results of the prediction in the foregroundregion wherein the pixel values change generally linearly with regard tothe pixel position.

[1372]FIG. 162A is a diagram which illustrates an example of the imagein the foreground region in the tutor image wherein the pixel valueschange generally linearly. FIG. 162B is a diagram which indicates changein pixel value corresponding to the position in the spatial direction inthe image in the foreground region of the tutor image wherein the pixelvalues change generally linearly.

[1373]FIG. 163A is a diagram which illustrates an example of the imagein the foreground region, corresponding to the image shown in FIG. 162A,generated by the conventional class classification adaptationprocessing. FIG. 163B is a diagram which indicates the change in pixelvalue corresponding to the position in the spatial direction, in theimage in the foreground region, corresponding to the image shown in FIG.162A, generated by the conventional class classification adaptationprocessing.

[1374]FIG. 164A is a diagram which illustrates an example of the imagein the foreground region corresponding to the image shown in FIG. 162A,generated by the region processing unit 5001 of which configuration isshown in FIG. 158. FIG. 164B is a diagram which indicates the change inpixel value, corresponding to the position in the spatial direction, inthe image in the foreground region, corresponding to the image shown inFIG. 162A, generated by the region processing unit 5001 of whichconfiguration is shown in FIG. 158.

[1375] The pixel values in the image in the foreground region generatedby the conventional class classification adaptation processing change ina stepped manner as compared with the tutor image in the same manner asthe mixed region, and the change in a stepped manner can be visuallyrecognized in the actual image.

[1376] Conversely, the pixel values in the image in the foregroundregion generated by the region processing unit 5001 of whichconfiguration is shown in FIG. 158, change more smoothly as comparedwith conventional arrangement, and are extremely close to the values inthe tutor image. In visual confirmation of the image generated by theregion processing unit 5001, difference between the image and the tutorimage could not be observed.

[1377] Also, the SN ratio at each of the regions of the image generatedby conventional class classification adaptation processing and the SNratio at each of the regions of the image generated by classclassification adaptation processing with the image processing deviceaccording to the present invention are obtained and compared.

[1378] The SN ratio in the covered background region of the imagegenerated by conventional class classification adaptation processing was32.1716 dB, the SN ratio in the uncovered background region was 31.8744dB, the SN ratio in the foreground region was 31.8835 dB, and the SNratio in the background region was 31.9985 dB.

[1379] Conversely, the SN ratio in the covered background region of theimage generated by the image processing device according to the presentinvention was 32.1799 dB, the SN ratio in the uncovered backgroundregion was 31.8922 dB, the SN ratio in the foreground region was 32.0925dB, and the SN ratio in the background region was 32.0177 dB.

[1380] In this way, the SN ratio in images generated by the imageprocessing device according to the present invention were higher thanthe SN ratio of images generated by conventional class classificationadaptation processing in all of the regions.

[1381]FIG. 165 is a flowchart explaining the processing of images withthe image processing device of which configuration is shown in FIG. 155.

[1382] In Step S5001, the region specifying unit 103 specifies theforeground region, background region, covered background region, anduncovered background region, in the input image, based on the movementvector and the positional information thereof supplied from the movementdetecting unit 102. The processing in Step S5001 is the same as theprocessing in Step S101, so description thereof will be omitted.

[1383] In Step S5002, the region processing unit 5001 divides the inputimage into the foreground region, background region, covered backgroundregion, and uncovered background region, which have been specified, andexecutes image processing for the foreground region, background region,and covered background region, and uncovered background region, whichhave been divided, and the processing ends.

[1384] In this way, the image processing device according to the presentinvention divides the input image into the foreground region, backgroundregion, covered background region, and uncovered background region, andexecutes image processing for each of the foreground region, backgroundregion, covered background region, and uncovered background region,which have been divided.

[1385] The learning processing for generating coefficient sets used forprediction of pixel values in the class classification adaptationprocessing by the region processing unit 5001, of which configuration isshown in FIG. 157, will be described with reference to the flowchartshown in FIG. 166.

[1386] In Step S5101, the region dividing unit 5102 divides the tutorimage stored in the tutor image frame memory 5101, based on the regioninformation supplied from the region specifying unit 103. That is, theregion dividing unit 5102 supplies images made up of pixels belonging tothe background region of the tutor image, which is the divided tutorimage, to background region tutor image frame memory 5103. The regiondividing unit 5102 supplies images made up of pixels belonging to theuncovered background region of the tutor image, which is the dividedtutor image to uncovered background region tutor image frame memory5104.

[1387] The region dividing unit 5102 supplies images made up of pixelsbelonging to the covered background region of the tutor which is thedivided tutor image to covered background region tutor image framememory 5105. The region dividing unit 5102 supplies images made up ofpixels belonging to the foreground region of the tutor which is thedivided tutor image to foreground region tutor image frame memory 5106.

[1388] In Step S5102, weighted averaging units 5107-1 through 5107-4generate student images for the background region, foreground region,uncovered background region, and covered background region. That is tosay, the weighted averaging unit 5107-1 performs one-quarter weightedaveraging, for example, on the image made up of pixels belonging to thebackground region of the tutor image, stored in background region tutorimage frame memory 5103, so as to generate a student image correspondingto the image made up of pixels belonging to the background region of thetutor image. The weighted averaging unit 5107-2 performs one-quarterweighted averaging, for example, on the image made up of pixelsbelonging to the uncovered background region of the tutor image, storedin uncovered background region tutor image frame memory 5104, so as togenerate a student image corresponding to the image made up of pixelsbelonging to the uncovered background region of the tutor image.

[1389] The weighted averaging unit 5107-3 performs one-quarter weightedaveraging, for example, on the image made up of pixels belonging to thecovered background region of the tutor image, stored in coveredbackground region tutor image frame memory 5105, so as to generate animage made up of pixels belonging to the covered background region ofthe tutor image. The weighted averaging unit 5107-4 performs one-quarterweighted averaging, for example, on the image made up of pixelsbelonging to the foreground region of the tutor image, stored inforeground region tutor image frame memory 5106, so as to generate astudent image corresponding to the image made up of pixels belonging tothe foreground region of the tutor image.

[1390] In Step S5103, the learning unit 5112-1 generates a coefficientset corresponding to the background region, based on the image made upof pixels belonging to the background region of the tutor image storedin the background region tutor image frame memory 5103 and the studentimage corresponding to the image made up of pixels belonging to thebackground region of the tutor image stored in the background regionstudent image frame memory 5108.

[1391] In Step S5104, the learning unit 5112-2 generates a coefficientset corresponding to the uncovered background region, based on the imagemade up of pixels belonging to the uncovered background region of thetutor image stored in the uncovered background region tutor image framememory 5104 and the student image corresponding to the image made up ofpixels belonging to the uncovered background region of the tutor imagestored in the uncovered background region student image frame memory5109.

[1392] In Step S5105, the learning unit 5112-3 generates a coefficientset corresponding to the covered background region, based on the imagemade up of pixels belonging to the covered background region of thetutor image stored in the covered background region tutor image framememory 5105 and the student image corresponding to the image made up ofpixels belonging to the covered background region of the tutor imagestored in the covered background region student image frame memory 5110.

[1393] In Step S5106, the learning unit 5112-4 generates a coefficientset corresponding to the foreground region, based on the image made upof pixels belonging to the foreground region of the tutor image storedin the foreground region tutor image frame memory 5106 and the studentimage corresponding to the image made up of pixels belonging to theforeground region of the tutor image stored in the foreground regionstudent image frame memory 5111.

[1394] In Step S5107, the learning units 5112-1 through 5112-4 supplythe coefficient set corresponding to the background region, thecoefficient set corresponding to the uncovered background region, thecoefficient set corresponding to the covered background region, and thecoefficient set corresponding to the foreground region, to thecoefficient set memory 5113. The coefficient set memory 5113 stores thecoefficient sets corresponding to each of the background region, theforeground region, the uncovered background region, and the coveredbackground region, and the processing ends.

[1395] In this way, the region processing unit 5001 of whichconfiguration is shown in FIG. 157 can generate coefficient setscorresponding to the background region, coefficient sets correspondingto the uncovered background region, coefficient sets corresponding tothe covered background region, and coefficient sets corresponding to theforeground region.

[1396] The details of processing in Step S5103 through Step S5106 arethe same as the processing described with reference to the flowchartshown in FIG. 103, and accordingly, description thereof will be omitted.

[1397] Further, it is needless to say that the processing in Step S5103through Step S5106 can be performed in a serial manner or in a parallelmanner.

[1398] Next, the processing for creating images with the regionprocessing unit 5001 of which configuration is shown in FIG. 158 will bedescribed with reference to the flowchart shown in FIG. 167.

[1399] In Step S5201, the region dividing unit 5202 divides the inputimage into the background region, foreground region, covered backgroundregion, or uncovered background region, based on the region informationsupplied from the region specifying unit 103. That is to say, the regiondividing unit 5202 takes the divided input image and supplies an imagemade up of pixels belonging to the background region to the backgroundregion frame memory 5203, supplies an image made up of pixels belongingto the uncovered background region to the uncovered background regionframe memory 5204, supplies an image made up of pixels belonging to thecovered background region to the covered background region frame memory5205, and supplies an image made up of pixels belonging to theforeground region to the foreground region frame memory 5206.

[1400] In Step S5202, the mapping unit 5207-1 predicts an imagecorresponding to the image made up of pixels belonging to the backgroundregion stored in the background region frame memory 5203 by classclassification adaptation processing, based on the coefficient setcorresponding to the background region, stored in the coefficient setmemory 5208.

[1401] In Step S5203, the mapping unit 5207-2 predicts an imagecorresponding to the image made up of pixels belonging to the uncoveredbackground region stored in the uncovered background region frame memory5204 by class classification adaptation processing, based on thecoefficient set corresponding to the uncovered background region, storedin the coefficient set memory 5208.

[1402] In Step S5204, the mapping unit 5207-3 predicts an imagecorresponding to the image made up of pixels belonging to the coveredbackground region stored in the covered background region frame memory5205 by class classification adaptation processing, based on thecoefficient set corresponding to the covered background region, storedin the coefficient set memory 5208.

[1403] In Step S5205, the mapping unit 5207-4 predicts an imagecorresponding to the image made up of pixels belonging to the foregroundregion stored in the foreground region frame memory 5206 by classclassification adaptation processing, based on the coefficient setcorresponding to the foreground region, stored in the coefficient setmemory 5208.

[1404] In Step S5206, the synthesizing unit 5209 synthesizes thepredicted image corresponding to the image made up of pixels belongingto the background region, the predicted image corresponding to the imagemade up of pixels belonging to the uncovered background region, thepredicted image corresponding to the image made up of pixels belongingto the covered background region, and the predicted image correspondingto the foreground region. The synthesizing unit 5209 supplies thesynthesized image to frame memory 5210. The frame memory 5210 stores theimage supplied from the synthesizing unit 5209.

[1405] In Step S5207, the frame memory 5210 outputs the stored predictedimages output, and the processing ends.

[1406] In this way, the image processing device having the regionprocessing unit 5001 of which configuration is shown in FIG. 158 candivide input images for each of the background region, the uncoveredbackground region, the covered background region, and the foregroundregion, and generate predicted images for each divided image.

[1407] The details of processing in Step S5202 through Step S5205 arethe same as the processing described with reference to the flowchartshown in FIG. 109, and accordingly, description thereof will be omitted.

[1408] Further, it is needless to say that the processing in Step S5202through Step S5205 can be performed in a serial manner or in a parallelmanner.

[1409] Also, the processing executed by themovement-blurring-removed-image processing unit 108, the separated imageprocessing unit 4002, and the region processing unit 5001 is notrestricted to the generating of the coefficients corresponding to an SDimage and an HD image, or the processing for generating of an HD imagefrom an SD image, and an arrangement may be made wherein an even higherresolution image in the spatial direction is generated by generating thecoefficients for generating an even higher resolution image in thespatial direction, for example. Moreover, an arrangement may be madewherein the movement-blurring-removed-image processing unit 108, theseparated image processing unit 4002, and the region processing unit5001 perform processing for generating an even higher resolution imagein the time direction.

[1410] Note that an arrangement may be made wherein themovement-blurring-removed-image processing unit 108, the separated imageprocessing unit 4002, and the region processing unit 5001, generatecoefficients from predetermined information, and execute classclassification adaptation processing based on the generatedcoefficients.

[1411] Also, an arrangement may be made wherein themovement-blurring-removed-image processing unit 108, the separated imageprocessing unit 4002, and the region processing unit 5001 perform otherprocessing, e.g., image size conversion into a desired size, extractingof color signals such as RGB, noise removal, image compressing,encoding, etc., based on class classification processing. For example,the compression ratio can be increased with little deterioration of theimage over conventional arrangements, by themovement-blurring-removed-image processing unit 108, the separated imageprocessing unit 4002, and the region processing unit 5001, compressingimages of each of the regions with low compression ratio in directionsfollowing movement vectors and high compression ratio in directionsorthogonal to movement vectors, based on movement vectors correspondingto the classes into which classification has been made and each of theimages.

[1412]FIG. 168 is a block diagram which illustrates anotherconfiguration of the region processing unit 5001 for applying the edgeenhancement processing having different effects to each of thebackground region, the uncovered background region, the coveredbackground region, or the foreground region.

[1413] The frame memory 5501 stores the input image in increments offrames. The frame memory 5501 supplies the input image stored therein tothe region dividing unit 5502.

[1414] The region dividing unit 5502 divides the input image into thebackground region, foreground region, covered background region, oruncovered background region, based on the region information suppliedfrom the region specifying unit 103. That is to say, the region dividingunit 5502 takes the divided input image and supplies a background imagemade up of pixels belonging to the background region to the backgroundregion frame memory 5503, supplies an image made up of pixels belongingto the uncovered background region to the uncovered background regionframe memory 5504, supplies an image made up of pixels belonging to thecovered background region to the covered background region frame memory5505, and supplies a foreground image made up of pixels belonging to theforeground region to the foreground region frame memory 5506.

[1415] The background region frame memory 5503 stores the backgroundimage made up of pixels belonging to the background region, suppliedfrom the region dividing unit 5502. The background region frame memory5503 supplies the background image stored therein, to the edge enhancingunit 5507-1.

[1416] The uncovered background region frame memory 5504 stores theimage made up of pixels belonging to the uncovered background region,supplied from the region dividing unit 5502. The uncovered backgroundregion frame memory 5504 supplies the image made up of pixels belongingto the uncovered background region stored therein, to the edge enhancingunit 5507-2.

[1417] The covered background region frame memory 5505 stores the imagemade up of pixels belonging to the covered background region, suppliedfrom the region dividing unit 5502. The covered background region framememory 5505 supplies the image made up of pixels belonging to thecovered background region stored therein, to the edge enhancing unit5507-3.

[1418] The foreground region frame memory 5506 stores the foregroundimage made up of pixels belonging to the foreground region, suppliedfrom the region dividing unit 5502. The foreground region input imageframe memory 5506 supplies the foreground image stored therein, to theedge enhancing unit 5507-4.

[1419] The edge enhancing unit 5507-1 applies processing for edgeenhancement suitable for the background image, to the background imagestored in the background region frame memory 5503, and supplies thebackground image that has been subjected to edge enhancement to thesynthesizing unit 5508.

[1420] For example, the edge enhancing unit 5507-1 executes edgeenhancing processing for enhancing the edges of the background image,which is a still image, more than as compared with the uncoveredbackground region, covered background region, or foreground region.Thus, the sense-of-resolution of the background image can be improvedeven further, without causing unnatural image deterioration uponapplication of edge enhancement processing to moving images.

[1421] The edge enhancing unit 5507-2 applies edge enhancementprocessing suitable for the uncovered background image, to the imagestored in the uncovered background region frame memory 5504, andsupplies the image that has been subjected to edge enhancement to thesynthesizing unit 5508.

[1422] For example, the edge enhancing unit 5507-2 executes edgeenhancing processing for enhancing the edges of the uncovered backgroundimage, which is an image containing moving foreground components, to adegree less than as compared with the background region. Thus, unnaturalimage deterioration upon application of edge enhancement processing tomoving images can be reduced, while improving the sense-of-resolution,in uncovered background region images.

[1423] The edge enhancing unit 5507-3 applies edge enhancementprocessing suitable for the covered background region image, to theimage stored in the covered background region frame memory 5505, andsupplies the image that has been subjected to edge enhancement to thesynthesizing unit 5508.

[1424] For example, the edge enhancing unit 5507-3 executes edgeenhancing processing for enhancing the edges of the covered backgroundimage, which is an image containing moving foreground components, to adegree less than as compared with the background region. Thus, unnaturalimage deterioration upon application of edge enhancement processing tomoving images can be reduced, while improving the sense-of-resolution,in covered background region images.

[1425] The edge enhancing unit 5507-4 applies edge enhancementprocessing suitable for the foreground image, to the foreground imagestored in the foreground region frame memory 5506, and supplies theforeground image that has been subjected to edge enhancement to thesynthesizing unit 5508.

[1426] For example, the edge enhancing unit 5507-4 executes edgeenhancing processing for enhancing the edges of the moving foregroundimage to a degree of edge enhancing less than as compared with thebackground region. Thus, unnatural image deterioration upon applicationof edge enhancement processing to moving images can be reduced, whileimproving the sense-of-resolution, in foreground region images.

[1427] The synthesizing unit 5508 synthesizes the background imagesubjected to edge enhancement that has been supplied from the edgeenhancing unit 5507-1, the image made up of pixels belonging to theuncovered background region subjected to edge enhancement that has beensupplied from the edge enhancing unit 5507-2, the image made up ofpixels belonging to the covered background region subjected to edgeenhancement that has been supplied from the edge enhancing unit 5507-3,and the foreground image subjected to edge enhancement that has beensupplied from the edge enhancing unit 5507-4, and supplies thesynthesized image to frame memory 5509.

[1428] The frame memory 5509 stores the synthesized images supplied fromthe synthesizing unit 5508, and also outputs the stored images as outputimages.

[1429] In this way, the region processing unit 5001 of whichconfiguration is shown in FIG. 168 applies edge enhancement processingcorresponding to the nature of each image, for each of the backgroundregion, uncovered background region, covered background region, orforeground region, so the sense-of-resolution of the image can beimproved without unnatural deterioration of the image.

[1430] The edge enhancing unit 5507-1 through edge enhancing unit 5507-4have the same configuration as the edge enhancing unit 1203, andaccordingly description thereof will be omitted.

[1431]FIG. 169 is a diagram describing the processing of the regionprocessing unit 5001 of which configuration is shown in FIG. 168.

[1432] The foreground region, uncovered background region, coveredbackground region, and background region of the input image arespecified by the region specifying unit 103.

[1433] The input image regarding which regions have been specified isdivided into the regions by the region dividing unit 5001. Thebackground region image, uncovered background region image, coveredbackground region image, and foreground region image, which have beendivided, are each subjected to edge enhancement, for each image,corresponding to the nature of each image, by the region processing unit5001 of which configuration is shown in FIG. 168.

[1434] The background region image, uncovered background region image,covered background region image, and foreground region image, which haveeach been subjected to edge enhancement, are synthesized.

[1435]FIG. 170 is a flowchart describing the processing of images withthe image processing device of which configuration is shown in FIG. 115.

[1436] In Step S5501, the region specifying unit 103 specifies theforeground region, background region, covered background region, anduncovered background region of the input image, based on the movementvector and the positional information thereof supplied from the movementdetecting unit 102. The processing in Step S5501 is the same as theprocessing in Step S5001, so description thereof will be omitted.

[1437] In Step S5502, the region processing unit 5001 divides the inputimage into the foreground region, background region, covered backgroundregion, and uncovered background region, which have been specified, andexecutes edge enhancement processing of the images for the foregroundregion, background region, and covered background region, and uncoveredbackground region, which have been divided, and the processing ends.

[1438] In this way, the image processing device according to the presentinvention divides the input image into the foreground region, backgroundregion, covered background region, and uncovered background region, andexecutes edge enhancement processing for each of the foreground region,background region, covered background region, and uncovered backgroundregion, which have been divided.

[1439] Next, the processing for edge enhancement by the regionprocessing unit 5001 corresponding to Step S5502 will be described withreference to the flowchart in FIG. 171.

[1440] In Step S5521, the region dividing unit 5502 divides the inputimage into the background region, foreground region, covered backgroundregion, or uncovered background region, based on the region informationsupplied from the region specifying unit 103. That is to say, the regiondividing unit 5502 supplies background images made up of pixelsbelonging to the background region, which is the divided input image, tobackground region frame memory 5503, supplies images made up of pixelsbelonging to the uncovered background region to uncovered backgroundregion frame memory 5504, supplies images made up of pixels belonging tothe covered background region to covered background region frame memory5505, and supplies foreground images made up of pixels belonging to theforeground region to foreground region frame memory 5506.

[1441] In Step S5522, the edge enhancing unit 5507-1 performs edgeenhancement of the background image stored in the background regionframe memory 5503, by edge enhancement processing corresponding to thenature of the background image.

[1442] In Step S5523, the edge enhancing unit 5507-2 performs edgeenhancement of the uncovered background region image, stored in theuncovered background region frame memory 5504, by edge enhancementprocessing corresponding to the nature of the image of the uncoveredbackground region.

[1443] In Step S5524, the edge enhancing unit 5507-3 performs edgeenhancement of the covered background region image, stored in thecovered background region frame memory 5505, by edge enhancementprocessing corresponding to the nature of the image of the coveredbackground region.

[1444] In Step S5525, the edge enhancing unit 5507-4 performs edgeenhancement of the foreground image, stored in the foreground regionframe memory 5506, by edge enhancement processing corresponding to thenature of the foreground image.

[1445] In Step S5526, the synthesizing unit 5508 synthesizes thebackground image, the image of the uncovered background region, theimage of the covered background region, and the foreground image, whichare each subjected to edge enhancement. The synthesizing unit 5508supplies the synthesized image to the frame memory 5509. The framememory 5509 stores the images supplied from the synthesizing unit 5508.

[1446] In Step S5527, the frame memory 5509 outputs the synthesizedimage stored therein, and the processing ends.

[1447] In this way, the region processing unit 5001 of whichconfiguration is shown in FIG. 168 applies edge enhancement processingcorresponding to the nature or each image, for each of the backgroundimage, uncovered background region image, covered background regionimage, and foreground image, so the sense-of-resolution of the image canbe improved without unnatural distortion in moving images.

[1448] Note that it is needless to say that the processing in Step S5522through Step S5525 can be performed in a serial manner or in a parallelmanner.

[1449]FIG. 172 is a diagram describing yet other processing of the imageprocessing device of which configuration is shown in FIG. 131.

[1450] As shown in FIG. 172, the input image has the foreground region,background region, and mixed region thereof specified, and is dividedinto the specified foreground region, background region, and mixedregion.

[1451] Processing for generating coefficients, or processing for noiseremoval, etc., for example, is applied to each of the image of theforeground region and the image of the background region which have beendivided.

[1452]FIG. 173 is a block diagram illustrating the configuration of aregion processing unit 5001 for generating coefficient sets used inclass classification adaptation processing for removing noise. Tutorimage frame memory 5701 stores input images in increments of frames. Thetutor image frame memory 5701 supplies the stored input images to aregion dividing unit 5702.

[1453] The region dividing unit 5702 divides the tutor image, which isthe input image, into the background region or foreground region, basedon the region information supplied from the region specifying unit 103.

[1454] The region processing unit 5001 shown in FIG. 173 uses neitherimages of the uncovered background region nor images of the coveredbackground region.

[1455] The region dividing unit 5702 supplies the background regionimage of the tutor image, which is the divided tutor image, tobackground region tutor image frame memory 5703, and supplies theforeground region image of the tutor image to foreground region tutorimage frame memory 5704.

[1456] The background region tutor image frame memory 5703 stores thebackground region image of the tutor image supplied from the regiondividing unit 5702. The background region tutor image frame memory 5703supplies the background region image of the tutor image stored thereinto a noise adding unit 5705-1 and a learning unit 5708-1.

[1457] The foreground region tutor image frame memory 5704 stores theforeground region image of the tutor image supplied from the regiondividing unit 5702. The foreground region tutor image frame memory 5704supplies the foreground region image of the tutor image stored thereinto a noise adding unit 5705-2 and a learning unit 5708-2.

[1458] The noise adding unit 5705-1 generates random numbers, forexample, and adds the random numbers to the pixels values of thebackground region image of the tutor image supplied from the backgroundregion tutor image frame memory 5703, thereby adding noise to the imageof the background region. The noise adding unit 5705-1 supplies thebackground region image to which noise has been added, to backgroundregion student image frame memory 5706.

[1459] The background region student image frame memory 5706 stores thebackground region image to which noise has been added which has beensupplied from the noise adding unit 5705-1, as a student image. Thebackground region student image frame memory 5706 supplies the studentimage corresponding to the background region image of the tutor imagestored therein, to the learning unit 5708-1.

[1460] The noise adding unit 5705-2 generates random numbers, forexample, and adds the random numbers to the pixels values of theforeground region image of the tutor image supplied from the foregroundregion tutor image frame memory 5704, thereby adding noise to the imageof the foreground region. The noise adding unit 5705-2 supplies theforeground region image to which noise has been added, to foregroundregion student image frame memory 5707.

[1461] The foreground region student image frame memory 5707 stores theforeground region image to which noise has been added which has beensupplied from the noise adding unit 5705-2, as a student image. Theforeground region student image frame memory 5707 supplies the studentimage corresponding to the foreground region image of the tutor imagestored therein, to the learning unit 5708-2.

[1462] The learning unit 5708-1 generates a coefficient setcorresponding to the background region, based on the tutor image whichis the image of the background region supplied from the backgroundregion tutor image frame memory 5703 and the student image to whichnoise has been added that is supplied from the background region studentimage frame memory 5706, and supplies the generated coefficient set tothe coefficient set memory 5709.

[1463] The learning unit 5708-2 generates a coefficient setcorresponding to the foreground region, based on the tutor image whichis the image of the foreground region supplied from the foregroundregion tutor image frame memory 5704 and the student image to whichnoise has been added that is supplied from the foreground region studentimage frame memory 5707, and supplies the generated coefficient set tothe coefficient set memory 5709.

[1464] The coefficient set memory 5709 stores the coefficient setcorresponding to the background region supplied from the learning unit5708-1, and the coefficient set corresponding to the foreground regionsupplied from the learning unit 5708-2.

[1465]FIG. 174 is a diagram describing the coefficient sets which theregion processing unit 5001 of which configuration is shown in FIG. 173generates. The region processing unit 5001 individually calculatescoefficient sets corresponding to the background region and coefficientsets corresponding to the foreground region. The region processing unit5001 does not generate coefficient sets corresponding to the uncoveredbackground region or the covered background region.

[1466] That is to say, the region dividing unit 5702 divides the inputimage into an image of the background region, an image made up of pixelsbelonging to the uncovered background region, an image made up of pixelsbelonging to the covered background region, and an image of theforeground region.

[1467] The learning unit 5708-1 calculates a coefficient setcorresponding to the background region, based on the divided backgroundregion image, and the learning unit 5708-2 calculates a coefficient setcorresponding to the foreground region, based on the divided foregroundregion image.

[1468] The coefficient set corresponding to the background region isused for prediction of pixel values of the background region in theclass classification adaptation processing for predicting pixel valuesfrom which noise has been removed. The coefficient set corresponding tothe foreground region is used for prediction of pixel values of theforeground region in the class classification adaptation processing forpredicting pixel values from which noise has been removed.

[1469] The predicted image corresponding to the image of the backgroundregion, the image corresponding to the uncovered background region, theimage corresponding to the covered background region, and the predictedimage corresponding to the image of the foreground region, aresynthesized into one image.

[1470] The learning processing for generating coefficient sets used forprediction of pixel values in the class classification adaptationprocessing by the region processing unit 5001 of which configuration isshown in FIG. 173 will be described with reference to the flowchart inFIG. 175.

[1471] In Step S5701, the region dividing unit 5702 region divides thetutor image stored in the tutor image frame memory 5701, based on theregion information supplied from the region specifying unit 103. That isto say, the region dividing unit 5702 supplies the background regionimage of the tutor image, which is the region-divided tutor image, tobackground region tutor image frame memory 5703. The region dividingunit 5702 supplies the foreground region image of the tutor image, whichis the region-divided tutor image, to foreground region tutor imageframe memory 5704.

[1472] In Step S5702, the noise adding units 5705-1 and 5705-2 generatestudent images for both the background region and the foreground region.That is to say, the noise adding unit 5705-1 generates random numbers,for example, and adds the random numbers to the pixels values of thebackground region image stored in the background region tutor imageframe memory 5703, thereby adding noise to the image of the backgroundregion. The noise adding unit 5705-2 generates random numbers, and addsthe random numbers to the pixels values of the foreground region imagestored in the foreground region tutor image frame memory 5704, therebyadding noise to the image of the foreground region.

[1473] In Step S5703, the learning unit 5708-1 generates a coefficientset corresponding to the background region, based on the tutor imagewhich is the image of the background region stored in the backgroundregion tutor image frame memory 5703 and the student image to whichnoise has been added that is stored in the background region studentimage frame memory 5706. The details of the processing for generatingcoefficient sets is the same as the processing that has been describedwith reference to the flowchart shown in FIG. 103, so descriptionthereof will be omitted.

[1474] In Step S5704, the learning unit 5708-2 generates a coefficientset corresponding to the foreground region, based on the tutor imagewhich is the image of the foreground region stored in the foregroundregion tutor image frame memory 5704 and the student image to whichnoise has been added that is stored in the foreground region studentimage frame memory 5707. The details of the processing for generatingcoefficient sets is the same as the processing that has been describedwith reference to the flowchart shown in FIG. 103, so descriptionthereof will be omitted.

[1475] In Step S5705, the learning units 5708-1 and 5708-2 supply thecoefficient set corresponding to the background region and thecoefficient set corresponding to the foreground region, respectively, tothe coefficient set memory 5709. The coefficient set memory 5709 storesthe coefficient sets corresponding to the background region and theforeground region, respectively, and the processing ends.

[1476] In this way, the region processing unit 5001 of whichconfiguration is shown in FIG. 173 can generate coefficient setscorresponding to the background region and coefficient setscorresponding to the foreground region.

[1477] Note that it is needless to say that the processing in Step S5703and Step S5704 can be performed in a serial manner or in a parallelmanner.

[1478]FIG. 176 is a block diagram illustrating the configuration of theregion processing unit 5001 for executing class classificationadaptation processing to generate images from which noise has beenremoved. Frame memory 5901 stores the input image in increments offrames. The frame memory 5901 supplies the input images stored thereinto the region dividing unit 5902.

[1479] The region dividing unit 5902 divides the input image into theforeground region, and background region, covered background region, oruncovered background region, based on the region information suppliedfrom the region specifying unit 103. That is to say, the region dividingunit 5902 supplies the background region image, which is the dividedinput image, to background region frame memory 5903, and supplies theforeground region image to foreground region frame memory 5906.

[1480] The region dividing unit 5902 supplies the covered backgroundregion and the uncovered background region image, which is the dividedinput image, to the synthesizing unit 5907.

[1481] The background region frame memory 5903 records the backgroundregion image made up of pixel belonging to the background regionsupplied from the region dividing unit 5902. The background region framememory 5903 supplies the stored background region image to the mappingunit 5905-1.

[1482] The foreground region frame memory 5904 records the foregroundregion image made up of pixel belonging to the foreground regionsupplied from the region dividing unit 5902. The foreground region inputimage frame memory 506 supplies the stored foreground region image tothe mapping unit 5905-2.

[1483] The mapping unit 5905-1 generates a predicted image from whichnoise has been removed, corresponding to the background region imagestored in the background region frame memory 5903 by classclassification adaptation processing based on the coefficient setcorresponding to the background region, stored in the coefficient setmemory 5906. The mapping unit 5905-1 supplies the generated predictedimage to the synthesizing unit 5907.

[1484] The mapping unit 5905-2 generates a predicted image from whichnoise has been removed, corresponding to the foreground region imagestored in the foreground region frame memory 5904 by classclassification adaptation processing based on the coefficient setcorresponding to the foreground region, stored in the coefficient setmemory 5906. The mapping unit 5905-2 supplies the generated predictedimage to the synthesizing unit 5907.

[1485] The synthesizing unit 5907 synthesizes the predicted imagecorresponding to the image of the background region supplied from themapping unit 5905-1, the predicted image corresponding to the image ofthe foreground region supplied from the mapping unit 5905-2, and theimages of the covered background region and uncovered background regionsupplied from the region dividing unit 5902, and supplies thesynthesized image to the frame memory 5908.

[1486] The frame memory 5908 stores the image supplied from thesynthesizing unit 5907, and also outputs the image stored therein as anoutput image.

[1487] Next, the processing for creating an image with the regionprocessing unit 5001 of which configuration is shown in FIG. 176, willbe described with reference to the flowchart shown in FIG. 177.

[1488] In Step S5901, the region dividing unit 5902 divides the inputimage into the background region, foreground region, covered backgroundregion, or uncovered background region, based on the region informationsupplied from the region specifying unit 103. That is to say, the regiondividing unit 5902 supplies the background region image, which is thedivided input image, made up of pixels belonging to the backgroundregion, to background region frame memory 5903, and supplies theforeground region image, made up of pixels belonging to the foregroundregion, to foreground region frame memory 5904.

[1489] The region dividing unit 5902 supplies the image made up ofpixels belonging to the covered background region and the image made upof pixels belonging to the uncovered background region, to thesynthesizing unit 5907.

[1490] In Step S5902, the mapping unit 5905-1 predicts an image fromwhich noise has been removed, corresponding to the background regionimage stored in the background region frame memory 5903 by classclassification adaptation processing based on the coefficient setcorresponding to the background region, stored in the coefficient setmemory 5906. The details of the processing for predicting images fromwhich noise has been removed is the same as the processing that has beendescribed with reference to the flowchart shown in FIG. 109, sodescription thereof will be omitted.

[1491] In Step S5903, the mapping unit 5905-2 predicts an image fromwhich noise has been removed, corresponding to the foreground regionimage stored in foreground region frame memory 5904 by classclassification adaptation processing based on the coefficient setcorresponding to the foreground region, stored in the coefficient setmemory 5906. The details of the processing for predicting images fromwhich noise has been removed is the same as the processing that has beendescribed with reference to the flowchart shown in FIG. 109, sodescription thereof will be omitted.

[1492] In Step S5904, the synthesizing unit 5907 synthesizes thepredicted image corresponding to the image of the background region, andthe predicted image corresponding to the image of the foreground region,and the image of the covered background region and the image of theuncovered background region. The synthesizing unit 5907 supplies thesynthesized image to the frame memory 5908. The frame memory 5908 storesthe image supplied from the synthesizing unit 5907.

[1493] In Step S5905, the frame memory 5908 outputs the storedsynthesized image, and the processing ends.

[1494] In this way, the image processing device comprising the regionprocessing unit 5001 of which configuration is shown in FIG. 176 candivide an input image into each of the background region, uncoveredbackground region, covered background region, and foreground region,generate predicted images for each of the divided background regionimage and foreground region image, and synthesize the generatedpredicted images with the images of the uncovered background region andcovered background region, so noise of the overall image can be reducedwithout unnatural degradation of images occurring at the mixed region.

[1495] Note that it is needless to say that the processing in Step S5902and Step S5903 can be performed in a serial manner or in a parallelmanner.

[1496] Also, with the present invention, image processing encompassesprocessing for allowing images to pass without change.

[1497]FIG. 178 is a block diagram illustrating yet another configurationof the functions of the image processing device according to the presentinvention.

[1498] Parts which are the same as the case shown in FIG. 11 are denotedwith the same reference numerals, and description thereof will beomitted.

[1499] Input images provided to the signal processing device areprovided to an object extracting unit 101, a region specifying unit 103,a mixture ratio calculation unit 104, and a foreground/backgroundseparation unit 105.

[1500] The object extracting unit 101 roughly extracts the image objectscorresponding to the foreground object contained in the input image, andsupplies the extracted image object to the movement detecting unit 102.The object extracting unit 101 roughly extracts the image objectcorresponding to the background object contained in the input image, andsupplies the extracted image object to the movement detecting unit 102.

[1501] The movement detecting unit 102 calculates the movement vectorsof the image object corresponding to the roughly extracted foregroundobjects by techniques such as block matching, gradation, phasecorrelation, and pixel recursion, or the like, and provides thecalculated movement vectors and movement vector position information(information for specifying the pixel positions corresponding to themovement vectors) to the region specifying unit 103.

[1502] The region specifying unit 103 classifies each pixels of an inputimage into one of the foreground region, the background region, or themixed region, and supplies region information to the mixture ratiocalculation unit 104 and the foreground/background separation unit 105.Details of the foreground region, the background region, or the mixedregion, will be described later.

[1503] The mixture ratio calculation unit 104 calculates the mixtureratio α corresponding to the pixels contained in the mixed region basedupon the input image and the region information supplied from the regionspecifying unit 103, and supplies the calculated mixture ratio to theforeground/background separation unit 105.

[1504] The foreground/background separation unit 105 separates the inputimages into foreground component images and background component imagesbased upon the region information supplied from the region specifyingunit 103 and the mixture ratio α supplied from the mixture ratiocalculation unit 104, and supplies foreground component images andbackground component images to a separated image processing unit 7001.

[1505] The separated image processing unit 7001 applies differentprocessing for each of the foreground component image and backgroundcomponent image. For example, the separated image processing unit 7001generates coefficients used in class classification adaptationprocessing for generating an even higher resolution image, based on thebackground component image.

[1506] For example, the separated image processing unit 7001 appliesclass classification adaptation processing to the background componentimage, so as to create an image with even higher resolution, and alsoapplies linear interpolation processing to the foreground componentimage, thereby generating an image.

[1507] Also, the separated image processing unit 7001 applies edgeenhancement processing to only the background component image, andallows the foreground component image to pass as it is.

[1508]FIG. 179 is a flowchart describing the processing of images withthe image processing device according to the present invention.

[1509] In Step S7001, the region specifying unit 103 specifies theforeground region, background region, covered background region, anduncovered background region, based on the movement vectors and theposition information thereof, supplied from the movement detecting unit102. The processing in Step S7001 is the same as the processing in StepS101, so description thereof will be omitted.

[1510] In Step S7002, the mixture ratio calculation unit 104 calculatesthe mixture ratio α based upon the input image and the regioninformation supplied from the region specifying unit 103. The processingin Step S7002 is the same as the processing in Step S102, andaccordingly description thereof will be omitted.

[1511] In Step S7003, the foreground/background separation unit 105separates the input images into foreground component images made up offoreground components and background component images made up ofbackground components based upon the region information supplied fromthe region specifying unit 103 and the mixture ratio α supplied from themixture ratio calculation unit 104. The processing in Step S7003 is thesame as the processing in Step S103, and accordingly description thereofwill be omitted.

[1512] In Step S7004, the separated image processing unit 7001 executesprocessing for each of the foreground component image and backgroundcomponent image, and the processing ends. Details of the imageprocessing which the separated image processing unit 7001 executes willbe described later.

[1513] In this way, the image processing device according to the presentinvention separates the input image into foreground component image andbackground component image, and executes image processing on theseparated foreground component image and background component image.

[1514]FIG. 180 is a block diagram illustrating the configuration of theseparated image processing unit 7001 for generating a coefficient setwhich is used in class classification adaptation processing forgenerating an even higher resolution image in the spatial direction,corresponding to the background component image. For example, theseparated image processing unit 7001 of which the configuration is shownin FIG. 180 generates a coefficient set which is used in classclassification adaptation processing for generating HD images from SDimages based upon input SD images corresponding to the background image.

[1515] The separated image processing unit 7001 of which configurationis shown in FIG. 180 does not use foreground component images.

[1516] Tutor image frame memory 7021 stores the background componentimage supplied from the foreground/background separation unit 105. Thetutor image frame memory 7021 supplies the background component imagestored therein, i.e., the tutor image, to a weighted averaging unit 7022and a learning unit 7024.

[1517] The weighted averaging unit 7022 generates an SD image which is astudent image by one-quarter weighted-averaging of the backgroundcomponent image in the HD image, which is a tutor image, for example,supplied from the tutor image frame memory 7021, and supplies thegenerated SD image to student image frame memory 7023.

[1518] The student image frame memory 7023 stores the student imagecorresponding to the background component image which is the tutor imagesupplied from the weighted averaging unit 7022. The student image framememory 7023 supplies the stored student image to the learning unit 7024.

[1519] The learning unit 7024 generates a coefficient set correspondingto the background component image based upon the background componentimage which is the tutor image supplied from the tutor image framememory 7021 and the student image corresponding to the backgroundcomponent image supplied from the student image frame memory 7023, andsupplies the generated coefficient set to coefficient set memory 7025.

[1520] The coefficient set memory 7025 stores the coefficient setcorresponding to the background component image supplied from thelearning unit 7024.

[1521]FIG. 181 is a block diagram illustrating the configuration of thelearning unit 7024.

[1522] A class classification unit 7121 comprises a class tap obtainingunit 7131 and a waveform classification unit 7132, and classifies apixel of interest, which is a pixel at interest, of the input studentimage. The class tap obtaining unit 7131 obtains a predetermined numberof class taps which are pixels of the student image corresponding to thepixel of interest, and supplies the obtained class taps to the waveformclassification unit 7132.

[1523] The waveform classification unit 7132 performs classclassification processing wherein the input signals are classified intoseveral classes based upon the features thereof, and classifies a pixelof interest into one class, based on the class taps. For example, thewaveform classification unit 7132 classifies the pixel of interest intoone of 512 classes, and supplies the class No. corresponding to theclassified class to a prediction tap obtaining unit 7122.

[1524] The prediction tap obtaining unit 7122 obtains from the pixel ofthe student image the prediction tap which is an increment forcalculation of the predicted value of the original image (tutor image),corresponding to the class, based upon the class No., and supplies theobtained prediction tap and the class No. to a corresponding pixelobtaining unit 7123.

[1525] The corresponding pixel obtaining unit 7123 obtains pixel valuesof the pixels in the tutor image corresponding to the pixel values whichare to be predicted, based upon the prediction tap and the class No.,and supplies the prediction tap, the class No., and the obtained pixelvalues of the pixels in the tutor image corresponding to the pixelvalues which are to be predicted, to a normal equation generating unit7124.

[1526] The normal equation generating unit 7124 generates normalequations for calculating a coefficient set which is used in theadaptation processing, corresponding to the relationship between theprediction tap and the pixels which are to be predicted, based upon theprediction tap, the class No., and the obtained pixels which are to bepredicted, and supplies the generated normal equations to a coefficientcalculation unit 7125 along with the class No.

[1527] The coefficient calculation unit 7125 calculates a coefficientset which is used in the adaptation processing, corresponding to theclassified class, by solving the normal equations supplied from thenormal equation generating unit 7124. The coefficient calculation unit7125 supplies the calculated coefficient set to the coefficient setmemory 7025, along with the class No.

[1528] An arrangement may be made wherein the normal equation generatingunit 7124 generates a matrix corresponding to such normal equations, andthe coefficient calculation unit 7125 calculates a coefficient set basedupon the generated matrix.

[1529] The normal equation generating unit 7124 generates the normalequations for calculating the optimal prediction coefficients w for eachclass, and the coefficient calculation unit 7125 calculates theprediction coefficients w based upon the generated normal equations.

[1530] Also, the adaptation processing is not restricted to processingfor creating images with high resolution in the spatial direction, andmay be arranged to create images wherein the image resolution is thesame and noise is reduced, for example.

[1531] The learning processing for generating coefficient sets used forprediction of pixel values with class classification adaptationprocessing by the separated image processing unit 7001 of whichconfiguration is shown in FIG. 180 will be described with reference tothe flowchart shown in FIG. 182.

[1532] In Step S7021, the weighted averaging unit 7022 generates astudent image corresponding to the background component image which isthe tutor image, by one-quarter weighted-averaging of the backgroundcomponent image which is a tutor image, fore example, stored in thetutor image frame memory 7021.

[1533] In Step S7022, the learning unit 7024 generates a coefficient setcorresponding to the background component image based upon thebackground component image which is the tutor image stored in the tutorimage frame memory 7021 and the student image corresponding to thebackground component image stored in the student image frame memory7023, and supplies the generated coefficient set to the coefficient setmemory 7025. The coefficient set memory 7025 stores the coefficient setcorresponding to the background component image, and the processingends.

[1534] In this way, the separated image processing unit 7001 of whichconfiguration is shown in FIG. 180 can generate coefficient setscorresponding to the background component image.

[1535] Referring to the flowchart shown in FIG. 183, description willnow be made with regard to the processing for generating of acoefficient set corresponding to the background component image,performed by the learning unit 7024, corresponding to the processing inStep S7022.

[1536] In Step S7121, the learning unit 7024 judges whether or not thereare any unprocessed pixels in the student image corresponding to thebackground component image, and in the event that judgment is made thatthere are unprocessed pixels in the student image corresponding to thebackground component image, the flow proceeds to Step S7122, and thelearning unit 7024 obtains the pixel of interest from the student imagecorresponding to the background component image in raster scan sequence.

[1537] In Step S7123, the class tap obtaining unit 7131 of the classclassification unit 7121 obtains the class tap corresponding to thepixel of interest from the student image stored in the student imageframe memory 7023. In Step S7124, the waveform classification unit 7132of the class classification unit 7121 reduces the number of bits of thepixels making up the class tap by applying the ADRC processing to theclass tap, and performs class classification for the pixel of interest.In Step S7125, the prediction tap obtaining unit 7122 obtains theprediction tap corresponding to the pixel of interest from the studentimage stored in the student image frame memory 7023 based upon theclassified class.

[1538] In Step S7126, the corresponding pixel obtaining unit 7123obtains pixels corresponding to the pixel value which is to bepredicted, from the background component image which is a tutor imagestored in the tutor image frame memory 7021, based upon the classifiedclass.

[1539] In Step S7127, the normal equation generating unit 7124 addspixel values of the pixels corresponding to the prediction tap and thepixel value which is to be predicted, to the matrix for each class,based upon the class into which classification has been made, the flowreturns to Step S7121, and the learning unit 7024 repeats judgmentwhether or not there are any unprocessed pixels. The matrix for eachclass to which pixel values of the pixels corresponding to theprediction tap and the pixel value which is to be predicted are added,corresponds to the normal equation for calculating of the coefficientset for each class.

[1540] In Step S7121, in the event that judgment is made that there areno unprocessed pixels in the student image, the flow proceeds to StepS7128, and the normal equation generating unit 7124 supplies the matrixfor each class, wherein pixel values of the pixels corresponding to theprediction tap and the pixel value which is to be predicted have beenset, to the coefficient calculation unit 7125. The coefficientcalculation unit 7125 calculates the coefficient set for each classcorresponding to the background component image by solving the matrixfor each class wherein pixel values of the pixels corresponding to theprediction tap and the pixel value which is to be predicted have beenset.

[1541] Note that an arrangement may be made wherein the coefficientcalculation unit 7125 calculates the coefficient set for predicting ofthe pixel value by non-linear prediction, as well as by linearprediction.

[1542] In Step S7129, the coefficient calculation unit 7125 outputs thecoefficient set for each class corresponding to the background componentimage to the coefficient set memory 7025, and the processing ends.

[1543] As described above, the learning unit 7024 can generate acoefficient set corresponding to the background component image.

[1544] As described above, the separated image processing unit 7001 ofwhich configuration is shown in FIG. 180 can generate a coefficient setcorresponding to the background component image.

[1545]FIG. 184 is a block diagram which illustrates the configuration ofthe separated image processing unit 7001 for generating an even higherresolution image in the spatial direction by performing the classclassification adaptation processing for the background component image,and performing interpolation for the foreground component image. Forexample, the separated image processing unit 7001 of which configurationis shown in FIG. 184 generates an HD image by applying the classclassification adaptation processing to the background component imageof the input image which is an SD image, and applying interpolationprocessing to the foreground component image of the input image which isan SD image.

[1546] Frame memory 7301 stores the background component image suppliedfrom the separated image processing unit 105. The frame memory 7301supplies the stored background component image to a mapping unit 7302.

[1547] The mapping unit 7302 generates an predicted image correspondingto the background component image stored in the frame memory 7301 by theclass classification adaptation processing based upon the coefficientset corresponding to the background component image stored incoefficient set memory 7303. The mapping unit 7302 supplies thegenerated predicted image to frame memory 7304.

[1548] The frame memory 7304 stores the predicted image corresponding tothe stored background component image, and supplies the stored predictedimage to a synthesizing unit 7308.

[1549] Frame memory 7305 stores the foreground component image suppliedfrom the separated image processing unit 105. The frame memory 7305supplies the stored foreground component image to a linear interpolationprocessing unit 7306.

[1550] The linear interpolation processing unit 7306 generates an imagesubjected to interpolation corresponding to the foreground componentimage stored in the frame memory 7305 by linear interpolationprocessing.

[1551] For example, in the event of calculating the pixel value of thepixel Y₃₃(1) in an image which is to be subjected to interpolation,enclosed by a quadrangle in FIG. 97, the linear interpolation processingunit 7306 multiplies the corresponding predetermined weight to each ofpixels, i.e., X₂₂, X₂₃, X₂₄, X₃₂, X₃₃, X₃₄, X₄₂, X₄₃, X₄₄, and sets thesum of the products to the pixel value of the pixel Y₃₃(1). In the sameway, in the event of calculating the pixel value of the pixel Y₃₃(2) inan image which is to be interpolated, the linear interpolationprocessing unit 7306 multiplies the predetermined weight different fromthe case of calculating the pixel value of the pixel Y₃₃(1) to each ofthe pixels, i.e., X₂₂, X₂₃, X₂₄, X₃₂, X₃₃, X₃₄, X₄₂, X₄₃, X₄₄, and setsthe sum of the products to the pixel value of the pixel Y₃₃(2) Thelinear interpolation processing unit 7306 calculates the pixel values ofthe pixel Y₃₃(3) and the pixel Y₃₃(4) based upon the pixels X₂₂, X₂₃,X₂₄, X₃₂, X₃₃, X₃₄, X₄₂, X₄₃, X₄₄, by the same interpolation processing.

[1552] The linear interpolation processing unit 7306 supplies the imagesubjected to interpolation to frame memory 7307.

[1553] The frame memory 7307 stores the image subjected to interpolationbased upon the foreground component image, and supplies the stored imageto the synthesizing unit 7308.

[1554] The synthesizing unit 7308 synthesizes the predicted imagecorresponding to the background component image supplied from the framememory 7304 and the image which has been subjected to interpolationbased upon the foreground component image supplied from the frame memory7307, and outputs the synthesized image as an output image.

[1555]FIG. 185 is a block diagram which illustrates the configuration ofthe mapping unit 7302. A mapping unit 7321 comprises a classclassification unit 7331 for performing the class classificationprocessing, a prediction tap obtaining unit 7332 for performing theadaptation processing, and a prediction calculation unit 7333.

[1556] The class classification unit 7331 comprises a class tapobtaining unit 7351 and a waveform classification unit 7352, andperforms class classification processing for the pixel of interest inthe background component image.

[1557] The class tap obtaining unit 7351 obtains the predeterminednumber of class taps corresponding to the pixel of interest in the inputimage, and supplies the obtained class taps to the waveformclassification unit 7352. For example, the class tap obtaining unit 7351obtains nine class taps, and supplies the obtained class taps to thewaveform classification unit 7352.

[1558] The waveform classification unit 7352 reduces the number of bitsof pixels making up the class tap by applying the ADRC processing to theclass tap, classifies the pixel of interest into one of thepredetermined number of classes, and supplies the class No.corresponding to the classified class to the prediction tap obtainingunit 7332. For example, the waveform classification unit 7352 classifiesthe pixel of interest into one of 512 classes, and supplies the classNo. corresponding to the classified class to the prediction tapobtaining unit 7332.

[1559] The prediction tap obtaining unit 7332 obtains the predeterminednumber of prediction taps corresponding to the class from the inputimage based upon the class No., and supplies the obtained predictiontaps and the class No. to the prediction calculation unit 7333.

[1560] The prediction calculation unit 7333 obtains a coefficient setcorresponding to the class from the coefficient sets corresponding tothe background component image stored in the coefficient set memory 7303based upon the class No. The prediction calculation unit 7333 predictspixel values of the predicted image by linear prediction based upon thecoefficient set corresponding to the class, and the prediction tap. Theprediction calculation unit 7333 supplies the predicted pixel values tothe frame memory 7322.

[1561] Note that the prediction calculation unit 7333 may predict pixelvalues of the predicted image by non-linear prediction.

[1562] The frame memory 7322 stores the predicted pixel values suppliedfrom the mapping processing unit 7321, and outputs the image made up ofthe predicted pixel values.

[1563]FIG. 186 is a diagram which describes the processing by theseparated image processing unit 7001 of which configuration is shown inFIG. 184.

[1564] As shown in FIG. 186, the input image is divided into regions,and separated into the foreground components and the backgroundcomponents. The separated input image is synthesized into foregroundcomponent image and the background component image.

[1565] The separated image processing unit 7001 applies the classclassification adaptation processing to the separated backgroundcomponent image. The separated image processing unit 7001 applies linearinterpolation processing to the separated foreground component image.

[1566] Referring to the flowchart shown in FIG. 187, description willnow be made with regard to the processing for creation of an image bythe separated image processing unit 7001 of which configuration is shownin FIG. 184.

[1567] In Step S7301, the mapping unit 7302 predicts the imagecorresponding to the background component image stored in the framememory 7301 by the class classification adaptation processing based uponthe coefficient set corresponding to the background component imagestored in the coefficient set memory 7303. Details of the processing forprediction of the image corresponding to the background component imagewill be described later with reference to the flowchart shown in FIG.188.

[1568] The mapping unit 7302 supplies the predicted image correspondingto the background component image to the frame memory 7304. The framememory 7304 stores the predicted image corresponding to the backgroundcomponent image, and supplies the stored predicted image to thesynthesizing unit 7308.

[1569] In Step S7302, the linear interpolation processing unit 7306performs linear interpolation for the foreground component image storedin the frame memory 7305. The linear interpolation processing unit 7306supplies the image subjected to linear interpolation to the frame memory7307. The frame memory 7307 stores the image subjected to linearinterpolation, and supplies the stored image subjected to linearinterpolation to the synthesizing unit 7308.

[1570] In Step S7303, the synthesizing unit 7308 synthesizes thepredicted image corresponding to the background component image and theimage wherein the foreground component image has been subjected tolinear interpolation. The synthesizing unit 7308 outputs the storedsynthesized image, and the processing ends.

[1571] As described above, the image processing device having theseparated image processing unit 7001 of which configuration is shown inFIG. 184 can generate a predicted image corresponding to the backgroundcomponent image, individually generate images subjected to linearinterpolation corresponding to the foreground component image,synthesize the generated images, and output the synthesized image.

[1572] Note that it is needless to say that the processing in Step S7301and Step S7302 may be performed in parallel, as well as in serial.

[1573] Referring to the flowchart shown in FIG. 188, description will bemade with regard to the processing for prediction of the imagecorresponding to the background component image by the mapping unit 7302corresponding to Step S7301.

[1574] In Step S7321, the mapping unit 7302 judges whether or not thereare any unprocessed pixels in the background component image, and in theevent that judgment is made that there are unprocessed pixels in thebackground component image, the flow proceeds to Step S7322, and themapping processing unit 7321 obtains the coefficient set correspondingto the background component image stored in the coefficient set memory7303. In Step S7323, the mapping processing unit 7321 obtains the pixelof interest from the background component image stored in the framememory 7301 in raster scan sequence.

[1575] In Step S7324, the class tap obtaining unit 7351 of the classclassification unit 7331 obtains the class tap corresponding to thepixel of interest from the background component image stored in theframe memory 7301. In Step S7325, the waveform classification unit 7352of the class classification unit 7331 reduces the number of bits ofpixels making up the class tap by applying the ADRC processing to theclass tap, and performs class classification for the pixel of interest.In Step S7326, the prediction tap obtaining unit 7332 obtains theprediction tap corresponding to the pixel of interest from thebackground component image stored in the frame memory 7301 based uponthe classified class.

[1576] In Step S7327, the prediction computation unit 7333 predictspixel values of the predicted image by linear prediction based upon thecoefficient set corresponding to the background component image and theclassified class, and the prediction tap.

[1577] Note that the prediction computation unit 7333 may predict pixelvalues of the predicted image by non-linear prediction, as well as bylinear prediction.

[1578] In Step S7328, the prediction computation unit 7333 outputs thepredicted pixel value to the frame memory 7322. The frame memory 7322stores the pixel values supplied from the prediction calculation unit7333. The flow returns to Step S7321, and repeats judgment whether ornot there are any unprocessed pixels.

[1579] In Step S7321, in the event that judgment is made that there areno unprocessed pixels in the background component image, the flowproceeds to Step S7329, the frame memory 7322 outputs the predictedimage corresponding to the stored background component image, and theprocessing ends.

[1580] As described above, the mapping unit 7302 can predict the imagecorresponding to the background component image based upon thebackground component image.

[1581] As described above, the separated image processing unit 7001 ofwhich configuration is shown in FIG. 184 can generate a predicted imagecorresponding to the background component image, perform linearinterpolation for the foreground component image, and accordinglygenerate an image, wherein the resolution in the spatial direction hasbeen enhanced, without unnatural degradation occurring in the foregroundcomponent image containing movement blurring.

[1582]FIG. 189 is a block diagram which illustrates anotherconfiguration of the separated image processing unit 7001 for applyingedge enhancement processing for the background component image. Theseparated image processing unit 7001 of which configuration is shown inFIG. 189 applies edge enhancement processing to the background componentimage, and synthesizes the foreground component image as it is and thebackground component image to which edge enhancement processing has beenapplied.

[1583] The background component image supplied from theforeground/background separation unit 105 is input to an edge enhancingunit 7501, and the foreground component image supplied from theforeground/background separation unit 105 is input to a synthesizingunit 7502.

[1584] The edge enhancing unit 7501 applies edge enhancing processingsuitable to the background component image to the background componentimage supplied from the foreground/background separation unit 105, andsupplies the background component image subjected to edge enhancing tothe synthesizing unit 7502.

[1585] For example, the edge enhancing unit 7501 performs edge enhancingprocessing, which further enhances edges, for the background componentimage which is a still image. Thus, the sense-of-resolution of thebackground component image can be improved without unnatural degradationoccurring in the image in the event of applying the edge enhancingprocessing to the moving image.

[1586] Also, for example, in the event that the background moves, theedge enhancing unit 7501 performs edge enhancing processing, of whichdegree is less than as compared with the case wherein the backgroundkeeps still, on the background component image. Thus, thesense-of-resolution of the background component image can be furtherimproved without unnatural degradation of the image occurring in theevent of applying edge enhancing processing to the moving image.

[1587] The synthesizing unit 7502 synthesizes the background componentimage subjected to edge enhancing that has been supplied from the edgeenhancing unit 7501 and the foreground component image supplied from theforeground/background separation unit 105, and outputs the synthesizedimage.

[1588] As described above, the separated image processing unit 7001 ofwhich configuration is shown in FIG. 189 synthesizes the foregroundcomponent image as it is and the background component image to which theedge enhancing processing corresponding to the nature of the backgroundcomponent image has been applied, and accordingly, thesense-of-resolution of the image can be improved without unnaturaldegradation of the image occurring.

[1589]FIG. 190 is a block diagram which illustrates the configuration ofthe edge enhancing unit 7501. The background component image is input toa high pass filter 7521 and an addition unit 7523.

[1590] The high pass filter 7521 extracts components wherein the pixelvalue drastically changes with regard to the pixel position, so-calledhigh image frequency components, and removes components wherein thechange in pixel value is small with regard to the pixel position,so-called low image frequency components, from the background componentimage based upon the input filter coefficients and generates an edgeimage.

[1591] The high pass filter 7521 supplies the generated edge image to again adjustment unit 7522.

[1592] The gain adjustment unit 7522 amplifies or reduces the edge imagesupplied from the high pass filter 7521 based upon the input gainadjustment coefficient. In the event that the input gain adjustmentcoefficient is altered, the gain adjustment unit 7522 changes theamplification ratio (decay ratio) of the edge image. For example, in theevent of inputting the gain adjustment coefficient which designates theamplification ratio equal to or more than 1, the gain adjustment unit7522 amplifies the edge image, and in the event of inputting the gainadjustment coefficient which designates the amplification ratio lessthan 1, the gain adjustment unit 7522 reduces the edge image.

[1593] The gain adjustment unit 7522 supplies the edge image subjectedto gain adjustment to the addition unit 7523.

[1594] The addition unit 7523 adds the background component image andthe edge image subjected to gain adjustment, supplied from the gainadjustment unit 7522, and outputs the image subjected to addition.

[1595] As described above, the edge enhancing unit 7501 applies edgeenhancing processing to the background component image.

[1596]FIG. 191 is a block diagram which illustrates anotherconfiguration of the edge enhancing unit 7501. In an example shown inFIG. 191, the edge enhancing unit 7501 comprises a filter 7541.

[1597] The filter 7541 generates an edge-enhanced image by amplifyingcomponents wherein the pixel value drastically changes with regard tothe pixel position in the background component image, so-called highimage frequency components, based upon the input filter coefficients.

[1598] As described above, the edge enhancing unit 7501 performs edgeenhancing processing corresponding to the nature of the backgroundcomponent image based upon different filter coefficients or gainadjustment coefficients, for example.

[1599]FIG. 192 is a diagram which describes the processing by theseparated image processing unit 7001 of which configuration is shown inFIG. 189.

[1600] As shown in FIG. 192, the input image is divided into regions,and separated into the foreground components and the backgroundcomponents. The separated input image is synthesized into the foregroundcomponent image and the background component image.

[1601] The separated image processing unit 7001 applies edge enhancingprocessing to the separated background component image, and outputs thebackground component image subjected to edge enhancing. The separatedforeground component image is output as it is.

[1602] Referring to the flowchart shown in FIG. 193, the processing bythe separated image processing unit 7001 of which configuration is shownin FIG. 189 will now be described.

[1603] In Step S7501, the edge enhancing unit 7501 applies edgeenhancing processing to the background component image. The edgeenhancing unit 7501 supplies the background component image subjected toedge enhancing to the synthesizing unit 7502.

[1604] In Step S7502, the synthesizing unit 7502 synthesizes thebackground component image subjected to edge enhancing and theforeground component image supplied from the foreground/backgroundseparation unit 105, outputs the synthesized image, and the processingends.

[1605] As described above, the image processing device having theseparated image processing unit 7001 of which configuration is shown inFIG. 189 can perform edge enhancing for background component image,synthesize the background component image subjected to edge enhancingand the foreground component image as it is, and output the synthesizedimage, and accordingly, the image processing device can generate animage wherein the sense-of-resolution is improved without unnaturaldegradation occurring in the foreground component image containingmovement blurring.

[1606]FIG. 194 is a block diagram which further illustrates anotherconfiguration of the functions of the image processing device. While theimage processing device shown in FIG. 178 performs region specificationand calculation of the mixture ratio α sequentially, the imageprocessing device shown in FIG. 194 performs region specification andcalculation of the mixture ratio α in a parallel manner.

[1607] The same portions as the functions shown in the block diagram inFIG. 178 are denoted by the same reference numerals, and descriptionthereof will be omitted.

[1608] The input image is supplied to the object extracting unit 101,the region specifying unit 103, the mixture ratio calculation unit 3001,and the foreground/background separation unit 3002.

[1609] The mixture ratio calculation unit 3001 calculates the estimatedmixture ratio wherein an assumption is made that the pixel belongs tothe covered background region, and the estimated mixture ratio whereinan assumption is made that the pixel belongs to the uncovered backgroundregion, for each of pixels contained in the input image, based upon theinput image, and supplies the estimated mixture ratio wherein anassumption is made that the calculated pixel belongs to the coveredbackground region and the estimated mixture ratio wherein an assumptionis made that the pixel belongs to the uncovered background region, tothe foreground/background separation unit 3002.

[1610] As described above, the image processing device according to thepresent invention can perform processing for an image corresponding tothe mixture of the background image and the image of the moving object.

[1611] Also, the image processing device according to the presentinvention can sufficiently improve the sense-of-resolution without theimage which contains movement blurring becoming unnatural.

[1612] Note that while the movement of the object which is theforeground has been described as being from the left to the right, it isneedless to say that this is not restricted to that direction.

[1613] In the above, an example has been given of a case of projectingimages in real space having three-dimensional space and time-axisinformation onto time-space having two-dimensional space and time-axisinformation, using a video camera, but the present invention is notrestricted to this example, and may be applied to cases of projecting agreater amount of first information of a first dimension onto lesssecond information of a second dimension.

[1614] Note that the sensor is not restricted to a CCD, and may be asensor which is a solid-state image-taking device, e.g., a CMOS(Complementary Metal Oxide Semiconductor (complementary metal oxide filmsemiconductor)), BBD (Bucket Brigade Device), CID (Charge InjectionDevice), or CPD (Charge Priming Device) or the like, and is notrestricted to a sensor wherein detecting elements are arrayed in amatrix fashion, but may rather be a sensor wherein the detectingelements are arrayed in a row.

[1615] The recording medium storing the program for executing the signalprocessing of the present invention is not only configured of packagedmedia such as a magnetic disk 91 (including floppy (RegisteredTrademark) disks), optical disk 92 (including CD-ROMs (Compact Disc-ReadOnly Memory), DVDs (Digital Versatile Disc)), magneto-optical disk 93(including MDs (Mini-Disc) (Registered Trademark)), or semiconductormemory 94 or the like, storing the program, to be distributed separatelyfrom the computer as shown in FIG. 10 for providing the program tousers, but is configured of ROM 72 or a hard disk included in thestorage unit 78 or the like storing the program, provided to the user inthe state of being assembled into the computer beforehand.

[1616] Also, in the present Specification, the steps describing theprogram recorded in the recording medium includes processing which isexecuted in the time-sequence following the described order, of course,and also processing which is executed in parallel or individually, evenif not processed in time-sequence.

INDUSTRIAL APPLICABILITY

[1617] According to the first present invention, an image can beprocessed corresponding to the mixture of the background image and theimage of the moving object.

[1618] According to the second present invention, an image can beprocessed corresponding to the mixture of the background image and theimage of the moving object.

[1619] According to the third present invention, the sense-of-resolutioncan be sufficiently improved without the image which contains movementblurring becoming unnatural.

[1620] According to the fourth present invention, an image can beprocessed corresponding to the mixture of the background image and theimage of the moving object.

1. An image processing device for processing input image data made up ofa predetermined number of pieces of pixel data obtained by animage-taking device having a predetermined number of pixels havingtime-integration effects, said image processing device comprising:region specifying means for specifying, based on said input image data,a mixed region made up of a mixture of a foreground object componentconfiguring foreground objects and a background object componentconfiguring background objects, and a non-mixed region made up of one ofa foreground region made up of said foreground object component and abackground region made up of a background object component configuringsaid background objects, and outputting region specifying informationcorresponding to the results of specifying; and processing means forprocessing said input image data for each region specified by saidregion specifying information.
 2. An image processing device accordingto claim 1, wherein said processing means decide a class correspondingto each piece of pixel data of said input image data, corresponding tosaid region specifying information.
 3. An image processing deviceaccording to claim 1, wherein said processing means enhance the edges ofsaid input image data, corresponding to said region specifyinginformation.
 4. An image processing device according to claim 1, whereinsaid processing means process said pixel data of at least one region ofsaid mixed region and said non-mixed region.
 5. An image processingdevice according to claim 1, wherein said region specifying meansfurther specify a covered background region and an uncovered backgroundregion, and output region specifying information corresponding to theresults of specifying; and wherein said processing means further processsaid input image data for each of covered background region anduncovered background region.
 6. An image processing device according toclaim 1, wherein said processing means generate coefficients used inclass classification adaptation processing, for each region specified bysaid region specifying information.
 7. An image processing deviceaccording to claim 1, wherein said processing means generate outputimage data by class classification adaptation processing, for eachregion specified by said region specifying information.
 8. An imageprocessing device according to claim 1, wherein said processing meansenhance the edges of said input image data, for each region specified bysaid region specifying information.
 9. An image processing method forprocessing input image data made up of a predetermined number of piecesof pixel data obtained by an image-taking device having a predeterminednumber of pixels having time-integration effects, said methodcomprising: a region specifying step for specifying, based on said inputimage data, a mixed region made up of a mixture of a foreground objectcomponent configuring foreground objects and a background objectcomponent configuring background objects, and a non-mixed region made upof one of a foreground region made up of said foreground objectcomponent and a background region made up of a background objectcomponent configuring said background objects, and outputting regionspecifying information corresponding to the results of specifying; and aprocessing step for processing said input image data for each regionspecified by said region specifying information.
 10. An image processingmethod according to claim 9, wherein, in said processing step, a classcorresponding to each piece of pixel data of said input image data isdecided, corresponding to said region specifying information.
 11. Animage processing method according to claim 9, wherein, in saidprocessing step, the edges of said input image data are enhanced,corresponding to said region specifying information.
 12. An imageprocessing method according to claim 9, wherein, in said processingstep, said pixel data of at least one region of said mixed region andsaid non-mixed region is processed.
 13. An image processing methodaccording to claim 9, wherein, in said region specifying step, a coveredbackground region and an uncovered background region are furtherspecified, and region specifying information is output corresponding tothe results of specifying; and wherein, in said processing step, saidinput image data for each of covered background region and uncoveredbackground region is further processed.
 14. An image processing methodaccording to claim 9, wherein, in said processing step, coefficientsused in class classification adaptation processing are generated foreach region specified by said region specifying information.
 15. Animage processing method according to claim 9, wherein, in saidprocessing step, output image data is generated by class classificationadaptation processing for each region specified by said regionspecifying information.
 16. An image processing method according toclaim 9, wherein, in said processing step, the edges of said input imagedata are enhanced for each region specified by said region specifyinginformation.
 17. A recording medium storing a computer-readable programfor processing input image data made up of a predetermined number ofpieces of pixel data obtained by an image-taking device having apredetermined number of pixels having time-integration effects, saidprogram comprising: a region specifying step for specifying, based onsaid input image data, a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of saidforeground object component and a background region made up of abackground object component configuring said background objects, andoutputting region specifying information corresponding to the results ofspecifying; and a processing step for processing said input image datafor each region specified by said region specifying information.
 18. Arecording medium according to claim 17, wherein, in said processingstep, a class corresponding to each piece of pixel data of said inputimage data is decided, corresponding to said region specifyinginformation.
 19. A recording medium according to claim 17, wherein, insaid processing step, the edges of said input image data are enhanced,corresponding to said region specifying information.
 20. A recordingmedium according to claim 17, wherein, in said processing step, saidpixel data of at least one region of said mixed region and saidnon-mixed region is processed.
 21. A recording medium according to claim17, wherein, in said region specifying step, a covered background regionand an uncovered background region are further specified, and regionspecifying information is output corresponding to the results ofspecifying; and wherein, in said processing step, said input image datafor each of covered background region and uncovered background region isfurther processed.
 22. A recording medium according to claim 17,wherein, in said processing step, coefficients used in classclassification adaptation processing are generated for each regionspecified by said region specifying information.
 23. A recording mediumaccording to claim 17, wherein, in said processing step, output imagedata is generated by class classification adaptation processing for eachregion specified by said region specifying information.
 24. A recordingmedium according to claim 17, wherein, in said processing step, theedges of said input image data are enhanced for each region specified bysaid region specifying information.
 25. A program for causing acomputer, which processes input image data made up of a predeterminednumber of pieces of pixel data obtained by an image-taking device havinga predetermined number of pixels having time-integration effects, toexecute: a region specifying step for specifying, based on said inputimage data, a mixed region made up of a mixture of a foreground objectcomponent configuring foreground objects and a background objectcomponent configuring background objects, and a non-mixed region made upof one of a foreground region made up of said foreground objectcomponent and a background region made up of a background objectcomponent configuring said background objects, and outputting regionspecifying information corresponding to the results of specifying; and aprocessing step for processing said input image data for each regionspecified by said region specifying information.
 26. A program accordingto claim 25, wherein, in said processing step, a class corresponding toeach piece of pixel data of said input image data is decided,corresponding to said region specifying information.
 27. A programaccording to claim 25, wherein, in said processing step, the edges ofsaid input image data are enhanced, corresponding to said regionspecifying information.
 28. A program according to claim 25, wherein, insaid processing step, said pixel data of at least one region of saidmixed region and said non-mixed region is processed.
 29. A programaccording to claim 25, wherein, in said region specifying step, acovered background region and an uncovered background region are furtherspecified, and region specifying information is output corresponding tothe results of specifying; and wherein, in said processing step, saidinput image data for each of covered background region and uncoveredbackground region is further processed.
 30. A program according to claim25, wherein, in said processing step, coefficients used in classclassification adaptation processing are generated for each regionspecified by said region specifying information.
 31. A program accordingto claim 25, wherein, in said processing step, output image data isgenerated by class classification adaptation processing for each regionspecified by said region specifying information.
 32. A program accordingto claim 25, wherein, in said processing step, the edges of said inputimage data are enhanced for each region specified by said regionspecifying information.
 33. An image-taking device, comprising:image-taking means for outputting a subject image taken by animage-taking device having a predetermined number of pixels havingtime-integration effects as image data made up of a predetermined numberof pieces of pixel data; region specifying means for specifying, basedon said input image data, a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of saidforeground object component and a background region made up of abackground object component configuring said background objects, andoutputting region specifying information corresponding to the results ofspecifying; and processing means for processing said image data for eachregion specified by said region specifying information.
 34. Animage-taking device according to claim 33, wherein said processing meansdecide a class corresponding to each piece of pixel data of said imagedata, corresponding to said region specifying information.
 35. Animage-taking device according to claim 33, wherein said processing meansenhance the edges of said image data, corresponding to said regionspecifying information.
 36. An image-taking device according to claim33, wherein said processing means process said pixel data of at leastone region of said mixed region and said non-mixed region.
 37. Animage-taking device according to claim 33, wherein said regionspecifying means further specify a covered background region and anuncovered background region, and output region specifying informationcorresponding to the results of specifying; and wherein said processingmeans further process said image data for each of covered backgroundregion and uncovered background region.
 38. An image-taking deviceaccording to claim 33, wherein said processing means generatecoefficients used in class classification adaptation processing, foreach region specified by said region specifying information.
 39. Animage-taking device according to claim 33, wherein said processing meansgenerate output image data by class classification adaptationprocessing, for each region specified by said region specifyinginformation.
 40. An image-taking device according to claim 33, whereinsaid processing means enhance the edges of said image data, for eachregion specified by said region specifying information.
 41. An imageprocessing device for processing input image data made up of apredetermined number of pieces of pixel data obtained by an image-takingdevice having a predetermined number of pixels having time-integrationeffects, said image processing device comprising: region specifyingmeans for specifying, based on said input image data, a mixed regionmade up of a mixture of a foreground object component configuringforeground objects and a background object component configuringbackground objects, and a non-mixed region made up of one of aforeground region made up of said foreground object component and abackground region made up of a background object component configuringsaid background objects, and outputting region specifying informationcorresponding to the results of specifying; and class deciding means fordetermining classes corresponding to each set of pixel data of saidinput image data, corresponding to said region specifying information.42. An image processing device according to claim 41, wherein said classdeciding means decide a class corresponding to said pixel data of onlyregions which are a portion of said mixed region, said foregroundregion, and said background region.
 43. An image processing deviceaccording to claim 41, further comprising generating means forprocessing said pixel data of said input image data corresponding tosaid classes that have been decided, and generating coefficients used inclass classification adaptation processing.
 44. An image processingdevice according to claim 41, further comprising converting means forprocessing said pixel data of said input image data based on acoefficient for each of said classes, corresponding to said classes thathave been decided, and converting said input image data into outputimage data.
 45. An image processing device according to claim 41,wherein said region specifying means further specify a coveredbackground region and an uncovered background region, and output saidregion specifying information corresponding to the results ofspecifying; and wherein said class deciding means decide said classescorresponding to said pixel data of said input image data, correspondingto said covered background region or said uncovered background regionthat have been specified.
 46. An image processing method for processinginput image data made up of a predetermined number of pieces of pixeldata obtained by an image-taking device having a predetermined number ofpixels having time-integration effects, said method comprising: a regionspecifying step for specifying, based on said input image data, a mixedregion made up of a mixture of a foreground object component configuringforeground objects and a background object component configuringbackground objects, and a non-mixed region made up of one of aforeground region made up of said foreground object component and abackground region made up of a background object component configuringsaid background objects, and outputting region specifying informationcorresponding to the results of specifying; and a class deciding stepfor determining classes corresponding to each set of pixel data of saidinput image data, corresponding to said region specifying information.47. An image processing method according to claim 46, wherein, in saidclass deciding step, a class corresponding to said pixel data of onlyregions which are a portion of said mixed region, said foregroundregion, and said background region, is decided.
 48. An image processingmethod according to claim 46, further comprising a generating step forprocessing said pixel data of said input image data corresponding tosaid classes that have been decided, and generating coefficients used inclass classification adaptation processing.
 49. An image processingmethod according to claim 46, further comprising a converting step forprocessing said pixel data of said input image data based on acoefficient for each of said classes, corresponding to said classes thathave been decided, and converting said input image data into outputimage data.
 50. An image processing method according to claim 46,wherein, in said region specifying step, a covered background region andan uncovered background region are further specified, and said regionspecifying information is output corresponding to the results ofspecifying; and wherein, in said class deciding step, said classescorresponding to said pixel data of said input image data is decidedcorresponding to said covered background region or said uncoveredbackground region that have been specified.
 51. A recording mediumstoring a computer-readable program for processing input image data madeup of a predetermined number of pieces of pixel data obtained by animage-taking device having a predetermined number of pixels havingtime-integration effects, said program comprising: a region specifyingstep for specifying, based on said input image data, a mixed region madeup of a mixture of a foreground object component configuring foregroundobjects and a background object component configuring backgroundobjects, and a non-mixed region made up of one of a foreground regionmade up of said foreground object component and a background region madeup of a background object component configuring said background objects,and outputting region specifying information corresponding to theresults of specifying; and a class deciding step for determining classescorresponding to each set of pixel data of said input image data,corresponding to said region specifying information.
 52. A recordingmedium according to claim 51, wherein, in said class deciding step, aclass corresponding to said pixel data of only regions which are aportion of said mixed region, said foreground region, and saidbackground region, is decided.
 53. A recording medium according to claim51, said program further comprising a generating step for processingsaid pixel data of said input image data corresponding to said classesthat have been decided, and generating coefficients used in classclassification adaptation processing.
 54. A recording medium accordingto claim 51, said program further comprising a converting step forprocessing said pixel data of said input image data based on acoefficient for each of said classes, corresponding to said classes thathave been decided, and converting said input image data into outputimage data.
 55. A recording medium according to claim 51, wherein, insaid region specifying step, a covered background region and anuncovered background region are further specified, and said regionspecifying information is output corresponding to the results ofspecifying; and wherein, in said class deciding step, said classescorresponding to said pixel data of said input image data is decidedcorresponding to said covered background region or said uncoveredbackground region that have been specified.
 56. A program for causing acomputer, which processes input image data made up of a predeterminednumber of pieces of pixel data obtained by an image-taking device havinga predetermined number of pixels having time-integration effects, toexecute: a region specifying step for specifying, based on said inputimage data, a mixed region made up of a mixture of a foreground objectcomponent configuring foreground objects and a background objectcomponent configuring background objects, and a non-mixed region made upof one of a foreground region made up of said foreground objectcomponent and a background region made up of a background objectcomponent configuring said background objects, and outputting regionspecifying information corresponding to the results of specifying; and aclass deciding step for determining classes corresponding to each set ofpixel data of said input image data, corresponding to said regionspecifying information.
 57. A program according to claim 56, wherein, insaid class deciding step, a class corresponding to said pixel data ofonly regions which are a portion of said mixed region, said foregroundregion, and said background region, is decided.
 58. A program accordingto claim 56, further comprising a generating step for processing saidpixel data of said input image data corresponding to said classes thathave been decided, and generating coefficients used in classclassification adaptation processing.
 59. A program according to claim56, further comprising a converting step for processing said pixel dataof said input image data based on a coefficient for each of saidclasses, corresponding to said classes that have been decided, andconverting said input image data into output image data.
 60. A programaccording to claim 56, wherein, in said region specifying step, acovered background region and an uncovered background region are furtherspecified, and said region specifying information is outputcorresponding to the results of specifying; and wherein, in said classdeciding step, said classes corresponding to said pixel data of saidinput image data is decided corresponding to said covered backgroundregion or said uncovered background region that have been specified. 61.An image-taking device, comprising: image-taking means for outputting asubject image taken by an image-taking device having a predeterminednumber of pixels having time-integration effects as taken image datamade up of a predetermined number of pieces of pixel data; regionspecifying means for specifying, based on said taken image data, a mixedregion made up of a mixture of a foreground object component configuringforeground objects and a background object component configuringbackground objects, and a non-mixed region made up of one of aforeground region made up of said foreground object component and abackground region made up of a background object component configuringsaid background objects, and outputting region specifying informationcorresponding to the results of specifying; and class deciding means fordetermining classes corresponding to each set of pixel data of saidtaken image data, corresponding to said region specifying information.62. An image-taking device according to claim 61, wherein said classdeciding means decide a class corresponding to said pixel data of onlyregions which are a portion of said mixed region, said foregroundregion, and said background region.
 63. An image-taking device accordingto claim 61, further comprising generating means for processing saidpixel data of said taken image data corresponding to said classes thathave been decided, and generating coefficients used in classclassification adaptation processing.
 64. An image-taking deviceaccording to claim 61, further comprising converting means forprocessing said pixel data of said taken image data based on acoefficient for each of said classes, corresponding to said classes thathave been decided, and converting said taken image data into outputimage data.
 65. An image-taking device according to claim 61, whereinsaid region specifying means further specify a covered background regionand an uncovered background region, and output said region specifyinginformation corresponding to the results of specifying; and wherein saidclass deciding means decide said classes corresponding to said pixeldata of said taken image data, corresponding to said covered backgroundregion or said uncovered background region that have been specified. 66.An image processing device for processing input image data made up of apredetermined number of pieces of pixel data obtained by an image-takingdevice having a predetermined number of pixels having time-integrationeffects, said image processing device comprising: region specifyingmeans for specifying, based on said input image data, a mixed regionmade up of a mixture of a foreground object component configuringforeground objects and a background object component configuringbackground objects, and a non-mixed region made up of one of aforeground region made up of said foreground object component and abackground region made up of a background object component configuringsaid background objects, and outputting region specifying informationcorresponding to the results of specifying; and edge enhancing means forenhancing the edges of said input image data, corresponding to saidregion specifying information.
 67. An image processing device accordingto claim 66, wherein said region specifying means further specify acovered background region and an uncovered background region, and outputsaid region specifying information corresponding to the results ofspecifying; and wherein said edge enhancing means enhance the edges ofsaid input image data, corresponding to said covered background regionor said uncovered background region that have been specified.
 68. Animage processing method for processing input image data made up of apredetermined number of pieces of pixel data obtained by an image-takingdevice having a predetermined number of pixels having time-integrationeffects, said method comprising: a region specifying step forspecifying, based on said input image data, a mixed region made up of amixture of a foreground object component configuring foreground objectsand a background object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of saidforeground object component and a background region made up of abackground object component configuring said background objects, andoutputting region specifying information corresponding to the results ofspecifying; and an edge enhancing step for enhancing the edges of saidinput image data, corresponding to said region specifying information.69. An image processing method according to claim 68, wherein, in saidregion specifying step, a covered background region and an uncoveredbackground region are further specified, and said region specifyinginformation is output corresponding to the results of specifying; andwherein, in said edge enhancing step, the edges of said input image dataare enhanced corresponding to said covered background region or saiduncovered background region that have been specified.
 70. A recordingmedium storing a computer-readable program for processing input imagedata made up of a predetermined number of pieces of pixel data obtainedby an image-taking device having a predetermined number of pixels havingtime-integration effects, said program comprising: a region specifyingstep for specifying, based on said input image data, a mixed region madeup of a mixture of a foreground object component configuring foregroundobjects and a background object component configuring backgroundobjects, and a non-mixed region made up of one of a foreground regionmade up of said foreground object component and a background region madeup of a background object component configuring said background objects,and outputting region specifying information corresponding to theresults of specifying; and an edge enhancing step for enhancing theedges of said input image data, corresponding to said region specifyinginformation.
 71. A recording medium according to claim 70, wherein, insaid region specifying step, a covered background region and anuncovered background region are further specified, and said regionspecifying information is output corresponding to the results ofspecifying; and wherein, in said edge enhancing step, the edges of saidinput image data are enhanced corresponding to said covered backgroundregion or said uncovered background region that have been specified. 72.A program for causing a computer, which processes input image data madeup of a predetermined number of pieces of pixel data obtained by animage-taking device having a predetermined number of pixels havingtime-integration effects, to execute: a region specifying step forspecifying, based on said input image data, a mixed region made up of amixture of a foreground object component configuring foreground objectsand a background object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of saidforeground object component and a background region made up of abackground object component configuring said background objects, andoutputting region specifying information corresponding to the results ofspecifying; and an edge enhancing step for enhancing the edges of saidinput image data, corresponding to said region specifying information.73. A program according to claim 72, wherein, in said region specifyingstep, a covered background region and an uncovered background region arefurther specified, and said region specifying information is outputcorresponding to the results of specifying; and wherein, in said edgeenhancing step, the edges of said input image data are enhancedcorresponding to said covered background region or said uncoveredbackground region that have been specified.
 74. An image-taking device,comprising: image-taking means for outputting a subject image taken byan image-taking device having a predetermined number of pixels havingtime-integration effects as image data made up of a predetermined numberof pieces of pixel data; region specifying means for specifying, basedon said image data, a mixed region made up of a mixture of a foregroundobject component configuring foreground objects and a background objectcomponent configuring background objects, and a non-mixed region made upof one of a foreground region made up of said foreground objectcomponent and a background region made up of a background objectcomponent configuring said background objects, and outputting regionspecifying information corresponding to the results of specifying; andedge enhancing means for enhancing the edges of said image data,corresponding to said region specifying information.
 75. An image-takingdevice according to claim 74, wherein said region specifying meansfurther specify a covered background region and an uncovered backgroundregion, and output said region specifying information corresponding tothe results of specifying; and wherein said edge enhancing means enhancethe edges of said image data, corresponding to said covered backgroundregion or said uncovered background region that have been specified. 76.An image processing device for processing input image data made up of apredetermined number of pieces of pixel data obtained by an image-takingdevice having a predetermined number of pixels having time-integrationeffects, said image processing device comprising: region specifyingmeans for specifying, based on said input image data, at least one of amixed region made up of a mixture of a foreground object componentconfiguring foreground objects and a background object componentconfiguring background objects, and a non-mixed region made up of one ofa foreground region made up of said foreground object component and abackground region made up of a background object component configuringsaid background objects, and outputting region specifying informationcorresponding to the results of specifying; and processing means forprocessing said pixel data for at least one region of said mixed regionand said non-mixed region.
 77. An image processing device according toclaim 76, wherein said processing means process one region specified bysaid region specifying means with a method different from a method forprocessing the other region.
 78. An image processing device according toclaim 76, wherein said region specifying means further specify saidforeground region and said background region, and output said regionspecifying information corresponding to the results of specifying. 79.An image processing device according to claim 76, wherein said regionspecifying means further specify a covered background region and anuncovered background region, and output region specifying informationcorresponding to the results of specifying.
 80. An image processingdevice according to claim 76, further comprising separating means forseparating said pixel data of said mixed region into said foregroundobject component and said background object component, based on saidregion specifying information; wherein said processing means process atleast one of said foreground object component and said background objectcomponent.
 81. An image processing method for processing input imagedata made up of a predetermined number of pieces of pixel data obtainedby an image-taking device having a predetermined number of pixels havingtime-integration effects, said method comprising: a region specifyingstep for specifying, based on said input image data, at least one of amixed region made up of a mixture of a foreground object componentconfiguring foreground objects and a background object componentconfiguring background objects, and a non-mixed region made up of one ofa foreground region made up of said foreground object component and abackground region made up of a background object component configuringsaid background objects, and outputting region specifying informationcorresponding to the results of specifying; and a processing step forprocessing said pixel data for at least one region of said mixed regionand said non-mixed region.
 82. An image processing method according toclaim 81, wherein, in said processing step, one region specified by theprocessing in said region specifying step is processed with a methoddifferent from a method for processing the other region.
 83. An imageprocessing method according to claim 81, wherein, in said regionspecifying step, said foreground region and said background region arefurther specified, and said region specifying information is outputcorresponding to the results of specifying.
 84. An image processingmethod according to claim 81, wherein, in said region specifying step, acovered background region and an uncovered background region are furtherspecified, and region specifying information is output corresponding tothe results of specifying.
 85. An image processing method according toclaim 81, further comprising a separating step for separating said pixeldata of said mixed region into said foreground object component and saidbackground object component, based on said region specifyinginformation; wherein, in said processing step, at least one of saidforeground object component and said background object component areprocessed.
 86. A recording medium storing a computer-readable programfor processing input image data made up of a predetermined number ofpieces of pixel data obtained by an image-taking device having apredetermined number of pixels having time-integration effects, saidmethod comprising: a region specifying step for specifying, based onsaid input image data, at least one of a mixed region made up of amixture of a foreground object component configuring foreground objectsand a background object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of saidforeground object component and a background region made up of abackground object component configuring said background objects, andoutputting region specifying information corresponding to the results ofspecifying; and a processing step for processing said pixel data for atleast one region of said mixed region and said non-mixed region.
 87. Arecording medium according to claim 86, wherein, in said processingstep, one region specified by the processing in said region specifyingstep is processed with a method different from a method for processingthe other region.
 88. A recording medium according to claim 86, wherein,in said region specifying step, said foreground region and saidbackground region are further specified, and said region specifyinginformation is output corresponding to the results of specifying.
 89. Arecording medium according to claim 86, wherein, in said regionspecifying step, a covered background region and an uncovered backgroundregion are further specified, and region specifying information isoutput corresponding to the results of specifying.
 90. A recordingmedium according to claim 86, said program further comprising aseparating step for separating said pixel data of said mixed region intosaid foreground object component and said background object component,based on said region specifying information; wherein, in said processingstep, at least one of said foreground object component and saidbackground object component are processed.
 91. A program for causing acomputer, which processes input image data made up of a predeterminednumber of pieces of pixel data obtained by an image-taking device havinga predetermined number of pixels having time-integration effects, toexecute: a region specifying step for specifying, based on said inputimage data, at least one of a mixed region made up of a mixture of aforeground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of saidforeground object component and a background region made up of abackground object component configuring said background objects, andoutputting region specifying information corresponding to the results ofspecifying; and a processing step for processing said pixel data for atleast one region of said mixed region and said non-mixed region.
 92. Aprogram according to claim 91, wherein, in said processing step, oneregion specified by the processing in said region specifying step isprocessed with a method different from a method for processing the otherregion.
 93. A program according to claim 91, wherein, in said regionspecifying step, said foreground region and said background region arefurther specified, and said region specifying information is outputcorresponding to the results of specifying.
 94. A program according toclaim 91, wherein, in said region specifying step, a covered backgroundregion and an uncovered background region are further specified, andregion specifying information is output corresponding to the results ofspecifying.
 95. A program according to claim 91, further comprising aseparating step for separating said pixel data of said mixed region intosaid foreground object component and said background object component,based on said region specifying information; wherein, in said processingstep, at least one of said foreground object component and saidbackground object component are processed.
 96. An image-taking device,comprising: image-taking means for outputting a subject image taken byan image-taking device having a predetermined number of pixels havingtime-integration effects as image data made up of a predetermined numberof pieces of pixel data; region specifying means for specifying, basedon said image data, at least one of a mixed region made up of a mixtureof a foreground object component configuring foreground objects and abackground object component configuring background objects, and anon-mixed region made up of one of a foreground region made up of saidforeground object component and a background region made up of abackground object component configuring said background objects, andoutputting region specifying information corresponding to the results ofspecifying; and processing means for processing said pixel data for atleast one region of said mixed region and said non-mixed region.
 97. Animage-taking device according to claim 96, wherein said processing meansprocess one region specified by said region specifying means with amethod different from a method for processing the other region.
 98. Animage-taking device according to claim 96, wherein said regionspecifying means further specify said foreground region and saidbackground region, and output said region specifying informationcorresponding to the results of specifying.
 99. An image-taking deviceaccording to claim 96, wherein said region specifying means furtherspecify a covered background region and an uncovered background region,and output said region specifying information corresponding to theresults of specifying.
 100. An image-taking device according to claim96, further comprising separating means for separating said pixel dataof said mixed region into said foreground object component and saidbackground object component, based on said region specifyinginformation; wherein said processing means process at least one of saidforeground object component and said background object component.